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THE ASSESSMEOT OF QUALITY IN PH.D. PROGRAMS: A REPORT^ 
ON JUDGMENT^ BY GRADUATE DEANS 

Abstract 



Sixty-three graduate deans were selected to represent Ph*D,.- 
granting universities varied by geographic region, prestige, 
control ^ and size. Ratings on the importance of Ph.D. program 
characteristics to judgments about quality were obtained fr^m^^ 
more than 80 percent of the panel; in response to a seconds ques- 
tionnaire, panel t^iembers also rated the adequacy or acceptability 
of several alternative ways to measure each important character- 
istic. Procedures and results of the two questionnaires are 
discussed, a summary lis^ of endorsed program characteristics and 
indexes is presented, and further research is suggested. 
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High Quality is the goal of every Pl).D. program. Yet little 
is knoi||^,bout how to assess quality, or even what to assess. In 
an effb^r*^ to- narrow this knowledge gap, the Graduate Record 
Examinatibns Board together vWit^ the Council of Graduate^^ Schools 
aske(fd Educational Testing Service to identify program character-- 
istics which could be u^ed to assess quality in doctor^al education 
and to suggest some appropriate indicators of th^s'e character- • 
istics. This repjort presents preliminary result^ of a survey to 
selected graduate school Means which was undertaken in order to 
fulfill this request. Funds for the study were provided by the 
Graduate Record Examinations Board and Educational Testing Service. 

This preliminary report was prepared *for the September 1973 
meeting of the Graduate Record Examinations Board Research Com- 
mittee and for an October 1973 meeting pP the project steering" 
cpmmittee. In order to share the results more broadly, with a 
minimum of delay, this preliminary version is. also being distri- 
buted to participants in the study, members of the Council of 
Graduate Schools, and other, interested parties. Some further 
analysis of the jsecond survey resi^lts is anticipated and will be 
incorporated into subsequent articles or i;eports. In particular, 
14 late quesf^ionnaires will be added to the -tabulations in 
Appendix C so that these results reflect the opinions of 9Cr per? 
cent of the sampled graduate school deans, rather than the 
68 percent representation reflected^in this preliminary version 
of Appendix C. Other Suggestions for revision ar^ solicited 
from interested readers. 

Special appreciation is extended to the participant>6 in this' 
survey. With only one telephone reminder about three weeks after 
each questionnaire was mailed, 60 out of 63 panel members responded 
to. at least one of .the two requests for judgments. Responses on 
"^the questionnaires were consistently thoughtful and complete. Such 
expression of interest and cooperation is ext raordinary »f or a mail 
survey, particularly given the tight time requirements and the 
length and complexity of the questionnaires in this study. Whatever 
value may be found in the results is due to the high level of pro- 
fessional commitment and the good natures of these graduate deans, 
in addition to their expert judgment: about quality in doctoral 
education. 

Princeton, N. J . 
November, 197 3 

\ 
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y ^ the assessment of quality in' ph.d. programs: a report ' 

a\^ 



ON JUDGMENTS BY GRADUATE DEANS 



Mary Jcf Clark 



The purp6se of this project was to identify characteristics 
related to Ph.D. progrc^m quality and some acceptable Vays to. measure 
each characteristic by collecting Judgments from a sample of 
graduate school deans. Paft of ^ larger proposal to study the 
assessment of quality by developing in3truments anql .collectJitng 
data on programs in selected fields (Clar> & Hartnett, SL973), this • 
study was designed to provide four kinds of information: 

1) the extent of agreement among graduate %deans ^out Ph.D. 
program characteristics most important to judgments about 
quality ; 

.2) variations in ^the importance of characteristics depending 
upon the major purpose of a Ph.D. program; 

3) the judged adequacy or acceptability of possible measures 
for each' characteristic; and 

A) the currant* availability of various kinds of information. 

It was thought that these data* would be of immediate interest to 
members of the graduate school community, given recent pressures 
for program evaluatron at the gYaduate level, an^ also would help ^ 
\o specify areas for primary attention in the larger proposal for 
empirical research. " , 

The direction a^ scope' of the study .we^e determined in large 
part by a steering committee of graduate deans appointed by the 
Graduate Record Examinations Board and The Council of Graduate 
Schools.^ The project was funded by the GRE Board. 



Members of the steering committee appointed by GREB were William 
.Burke, Arizona State University; Bernard Harleston, Tufts University; 
'Robert MacFarland, University of Missouri at Rolla; and Donald Taylor, ^ 
Yale University. Members appointed by COS included iMary Evelyn Huey , 
Texas Woman's University; Philip Kubzansky, Boston University; Charles 
Lester, Emory University; and Joseph McCarthy, University of Washington, 
Michael Pelczar, chairman of the GREB, and J. Bpyxd Page, President of 
COS, were ex officio members of the committee. ^ 



Parameters or guidelines for the study, agreed upon by the 
steering committee at the outset, included the following: 

1. The survey would co'ncern Ph.D. programs only. . 

2. 'Threfe alternative purposes of^Ph.O.' programs ^would be 
^ considered: the training of scholarly researchers, 

^'^^^^oXJ^ege teachers, and professional practitioners. 

y 

^ if--.. 
*3. Dimeo,filons of quality t;hat are conlmon across many fields 
of study w(9uld be emphasized. 

4. An array of characteristics and meas^r^es would be sought, 
ratHer than a single rating or score. ' 

5. The survey would be limited tcv a small but varied sample 
of graduate school deans. 



•Procedure 




The study was designed as a two-stage mail survey to about 
60 graduate school deans. The first quest ionnaire sought opinions 
about the importance of various' program characteristics to the- 
achiev^iflent of high quality doctoral education for each of , three 
program goals. The second questionnaire suggested possible measui^e? 
for the more important program charaj: terist ics and asked for judgments 
about the adequacy and availability c^. each neasure. frequency 
tabulations were compiled on the importance of characteris^iics and 
on the adequacy of measures to assess the degree of Consensus among 
respondents on each variable. In addition, mean ratings were com- 
puted so that program characteristics could te arranged in the order 
of their rated importance and measures rated "good'! or better could 
be identified . Sample selection , ques^onnaire development , data 
collection, and data analysis are desct'ibed in greater detail in 
the following sections. 

Sample » 

Sixty universities offering the Ph.D. degree were selected for' 
Inclusion in the study, using the following criteria: V 

1. all universities represented on the GRE Board 

2. ^ all universities represented on the CGS executive committee 

'3. other universities to provide an over-all representat ive ^ 
sample of Ph.D. -granting universities by geographic region, 
prestige rating, control, and size. 
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f Th« sample was djfawn from the manter list of unlversltlvH Inclmiiid 
in the 1972 ORF./CCJS Cradimte PtogramB and Admissions Manual, which 
represents over 85 percent: of the graduate ervrollm<?;fit in. the ^ • 
Unlt<^d\state8 and an even hfgher percent' of the doctoral degrees 
awarded annually. These ^JuUverfjit ies were sorted by geographic' 
region, universities meetlng.the fir^t or second criterion wer^ 
identified, and additional universities were selected Trom each 
region to total about one out of every , four Plf.O, -gr??nt img 
universities ol varying pre:-tige, control, and size. The number 
of selected Institutions that fall In ga^ch^ -cell of the 'sampling 
matrix are reported In Table l\ Slightly over AO 'percent oT the 
univers It lejjj in tlu» sample* are Inclucjed In a dlst of '"top 50" 
Institutions (MLlln^an & TooPbs , 1972). About 60 percent of the 
universities are pu!/. ic and 5^) percent hav<» large, (more t»han 
2009 st^^ents) graduate<(pnro Hments , About 40 percent of the 
universities are pti vat e an^Jf '^'^ percent have small (less than 
2000 students) graduate enrcnlnients. It was assumed that the 
graduate deai s at /tliese varied kinds of Institutions would reflect, \^ 
at least in part, /the different philosophies and opinions about 
quality that mtgl^ be associated with each of them. Universities 
Included in the s^imple are ilsted in Appendix E. 

The final mailing list consisted of 15 members of the CRE Board, <) 
members of the CCS executive committee (not counting duplicates^ 
with the GREB), -and '39 graduate deans at other ' Inst icut Ions in the 
sample.. Beccuise of some overlap in categories , 'questionnaires were 
sent, to two persons at three universities for a total of 63 panel 
members. 

Though the sample was iUrawn to include as much diversity as 
-possible in a panel of 60 j/raduate deans, the results may not 
'accurately reflect the opinions of all graduate deans in every 
type of program. Also, theVudvantages of a . relat Ively small group 
for an-intenslve sarvey inevitably are balanced by the disadvantages 
of small numbers for interpretation and generalization of the results. 
These limitations of the .study should be kept in mind throughout the 
r^ainder of this report. 

(|u e s t i onna i r es 

The first questionnaire, concerning the Importance of program 
characteristics to lodgments about quality, grouped characteristics 
under four ma^or headin^'^s : Faculty Membe/s , St udents / Resources , and 
Program (^eratlon. Ton ko 20 specific characteristics were listed 
under each topic, such as "academic training'* and '^teaching experience" 
under. Faculty and ^'general academic ability^at entrart'ce" under Students 
Panel members were asked to indicate the importance of information # 
ahoij^t each charac t r i s c 1 c ver judging the educational quality o? 
a^Ph.D. pro-am to train a) researchers, b) teachers, and 





I 


Table 1 




r 






, ' Character i si i( s 


(;f Univers Itles in 


the Sample 






Geographic 

r c» gr in wf" 

\ 


f'p r «* 11 1 of" 
populat"ion 

2 

L n H ^rn nit* 


Pros t ige 


rat iiig 
o th t? r 


Control 
.pub lie private 


' llnder 
2000 


A * 

Over 

2000 


• 

Northeast 


AO ■ 


i 


5* 




• 


3 






7 




5 7 




6 


Midwest \ 




6 

« 


7 


10 3 - 


5 


8 


• South 


.-^"^ 


(» 


9 


lO • 5 


7 ' 


8 


West- 


29 , 


4 


8 


J- 3 


4 . 


8 • 


Total Number 


.. 60 


26 




36 . 24 


27 


. 33- 


Percent • 


/ 

27 


A3 


37 


60 40 


45 


55 



For composition of regions see Roose & Anderson, 197D> p» 5» 
"Population is. PH.D. universities included in the GREB/CGS Graduate 



Programs an^^/^A dmissions Manual, 197^. Thirty-seven states are 

represented ;ln the sample. 

* ■ 

3 - . 

Based on oomposite scgres prepared by J. W. Johnston from the Roose 
^nd Andersen ACE . rating§^ and reported in Millman & Toombs, 1972, p. 1,5. 

4 ' P- ' ■ ' " 
Total graduate school ensifcl^iment 1971-72 reported by the univer- 
sities and listed in the- TtAiruial . :- 
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c) pr^ict i t loners . Ratings were macU^ on a four point scal^ from " 
* es^i^it L.il" to "not important**. Respondents werf! invited to 
comment on 1 nd Iv idua 1 J tems *and tc^' add items if they felt important 
characteristics had been omitted, • ' 



Program charac t er i'st ics liiited under each heading on the first 
quest ionnalre' were identified through discussion with the project 
steering coinmlttee,, review of graduate education and evaluation'* 
literat\jre, and dis(russi'on. with others knowledgeable about graduate 
education. HelpfuT mater ials included Berelson, i960; Blackburn & 
■ Lingenf elter , 1972 ;- Brovnu 1970; dartter, 1966; Dres-sei,.et al,1970;' 
Elton & Kodgers, 1971; Elton 6. Rose, 1972; Hagstrom, 1971; Heiss, 1970; 
National Science Board, 1969; Powel 6. Lamson , 1972; Stanford, 1^72; 
a«d Tyler, 1972. I'rel iminary questionnaires from the Outcomes Iden- 
tification" pro j ect of the National Center for Higher Eduation 
Management Systems, Western Interstate Commission for Higher Education^ 
' al so were helpful , ' , ' / 

The first quest lonna ire also asked for certain background 

information about the respo^jdent , incLud ing his or ^her academic 

discipUr\e,. highest {legree , un ivers ity awarding highest degree, 'and 

years of experience on a graduate faculty. In addition, the deans 

were asked to rank ^t he major tasks of their oWn ins^titut ions ' Ph.D. 

programs, in the orde^r of their taportance in the physical sciences, 

h-iological sc'ienceis, social sciences, aTid ' humanit ies . The three 

major tasks to be ranked^ were the training of researching scholars, 

college tt^chers, and ' pro f ess ional practitioners. 

^ ^ . ' % • • • 

The second questionnaire built, on the results of the first one, 
ar rang ing • charac ter ist ics under each of tlie four headings in the order 
of their rated importance to quality and eliminating some of the 
'lower-rated c haracter i'Vit ics , As many as 13 possible measures were 
then listed for each c ha rac t c r l^ic , Many of the measures were 
suggesteii bv the literature reviewed to identify important program 
characteristics for the first questionnaire; additional helpful ^ ^ 
materials included Fenst emacher , 1972; Fleming, 1973; Gregg, 19i^; 
Lamson ^..Svaine, 1973 ; McMichael, 1973 ; Roaden, 1972; Roaden & 
Larimore, in press; Webb, 1 97 2; and guidelines for the 'evaluation 
of graduate education in the State University of New York, University 
of California at Berkeley, University of Washington, and University 
of Michigan. Si^me of these materials were supplied or suggested by 
panel members in re'spon^^* to the first questionnaire. 

The full list of possible measures for each charaet^rist ic may 
be found ia Appendix C, which presents the t^abulated results of 
the second questionnaire* > 

\ 



-13- 

ERIC 



- The second questionnaire a^^^^ panel members to respond in 
three ways to^each possible mea^^^®^' rat^ it^^ adequacy or 

appropriateness as an indicator quality of the listed 

program characteristic, using ^'^'^^ point scal^ from "very good" 
to "inadequate"; 2) indicate one treasure for each characteristic 
preferred by^the respondent; and ^) in'Ji^ate whether the informa- 
tion is current ly available, not available, or whether availability 
varies from program to prograrn. Again, respondents were invited 
to comment on each item and co ^^d measures to each list. 



Data Collection . . . / 

^Both questionnaires were nC^'^rnpanied by a personally addressed 
and typed cover letter signed by chairman of the Graduate Record 

Examinations Board and the presi^^^^ ^'ouncil of Graduate 

Schools. The first questionnaire mailed on May 3, 1973.- Fifty 

completed responses had been reC^^ved bv the cut-off date of June 4 . 
Four more were received too late ^e- counted, a total response 

rate of 86 percent and usable response rate of Sq percent. 

^ The characteristics ot ^esp^"*^^"^^ the first questionnaire 

are detailed in Table 2. Mo^t the graduate deans in the survey 
panel hold Ph . D il^degrees from pr^^^tiTg^ft. universities and have served 
for 20 years or more as a mem^ef ^ graduate school faculty. 
Almost three out of four of thern specialized in the social sciences, 

^p hysical sciences, or engin^ r in^.,^JL!]g-,g^££ia^i c regio ns, pr estige ^ 
control, and size of their emploY^^^ universities are very much like 
characteristics of the total sample (see Table i). Thus, though 
their own backgrounds tend to he Prestige iti^titutions and in 
the social or natural science^, *^^^>' ^^Pi^es^nt more diverse 

Institutions and types of gr^du^*^^ programs. 

The second questionnaire wa^ mailed on July 3^1973, with a 
cut-off date of July 31 for tabulation. Forty--three completed 
questionnaires had been received this time; nine more question- 
naires came in after this dat^, ^^^^ ^ tutal response rate of 83 percent 

The tight cut-off date for «^he second questionnaire, resulting in 
tabulation of 43 responses (68 percent of the p^nel) , was necessitated 
by deadlines to produce this rep^^^ September meeting of the 

GRE Board. Visual inspection of "^"^ ^^est ionnaires received 

after July 31 does not indicate ^^V glaring inconsistency ^^ith the 
43 making up the data pool, but ^^^^ could be determined for sure 
only by re-running the data with responses. Six of the 9 

uncounted questionnaires were fr^^ deans at large ''top 50" public 
in9^;itutions and these universities were somewhat underrepresented 
in the count. 5 ' 



Table 2 



Characteristics of Panel Members Who Returned 
the First Questionnaire (N==54^) 



Characteristics 



Personal Characteristics 



Title: Dean of the Graduate School 

Vice-president provost , or-chancellor 
Professor 

Assistant or associate dean 



Highest degree; 



Ph,D, 

Other doctorate 



Area of academic specialization; 
Humanit ies 
Social sciences 
Biological sciences 
Physical sciences & engineering 

University awarding highest degree: 
"Top 50" 
Other 



I 



39 


72 


8 . 


15 


3 


06 


A 


07 


49 


91 


5 


0^ 


6 


11 



20 
8 
20 



46 
8 



37 



85 
15 



Years of experience in graduate education: 
Ten years or less 
11 - 15 years 
16 - 20 years 
More th^n 20 years 
No arrswtrr ■ — 



4 
10 
12 

27 
- 1 



07 
19 
22 
50 
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Institutional Characteristics 



Geographic region: 
Nor theas t 
East 
Midwes t 
South 
West 



7 
10 
11 

14 
12 



13 
19 
20 
26 
22 



Prestige : 

"Top 50^ 
Other 



24 
30 



44 
56 



Control 



Public 
Private 



•34 
20 



63 

37 



Size : 



Fewer than 2,000 graduate students 
More than 2,000 graduate students 



26 
28 



48 

52 
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Data Tabulation i 

* Vhe f irst qyestionnaire was tabulated by hand for frequency 
distributions and 'mean ratings on the importance of information about 
various program characteristics in programs designed for three dif- 
ferent purposes. Through this procedure served its purpose of providing 
summary tabulations for use in construction of the second question- 
naire Immediately after the cut-off date,. wear and tear on personnel 
and prospects for error dictated machine tabulation for the second 
quest ionnaire . 2 Face-sheet information concerning the background and 
training of respondents, characteristics ot their universities, and 
rank order ratings of the major tasks of Ph.D. programs at their 
Upiverslties were keypunched and hand sorted by various subgroups 
of respondents. 



Results 



Detailed tabulations of the first questionnaire, concerning the 
importance of program characteristics, are presented in Appendix B; 
tabulated ratings of all suggested measures and their availability 
will be found in Appendix C. Appendix D summarizes the rank orders 
assigned by respondents to themajor tasks of their institutions' 
Ph.D. programs in four major curricular areas. Appendix A summarizes 
the materials in Appendixes B and C by listing only the most important 
program characteristics and the most acceptable measures in Che order ' 
of their endorsement by graduate deans. Though probably still some- 
what too long and detailed, this suiDmarv list of program character- 
istics and measures provides an important first step in efforts to 
establish a systematic procedure for the assessment of quality in 
Ph.D. programs of study. 

The remainder oA^this section briefly discusses the results! of 
each questionnaire and the summary listing of program characteristics 
and measures. 




First Questionnaire: Pr ogram Charact eristics Important to Qua 1 ity 

From detailed results of the first questionnaire (Appendix B) , 
it is apparent that graduate deans agreed about the importance of 
some kinds of program Information for the assessment of quality 



Robert A. Marozsan prepared the computer program for tabulation 
of the second questionnaire. 



and disagreed about the import^ance of 'other characteristics. In 
addition, about two-thirds of them made distinctions about the level 
of importance of certain characteristics depending upon whether 
the program was designed to train research scholars, college 
teachers, or professional practitioners. The greatest number of 
differences occurred in relation to the characteristics of faculty 
members (Table B.l). For instance, the research conducted by faculty 
members .and the nat ional ' honors they receive were considered very 
important to quality, in programs designed to train research scholars. 
But teaching effectiveness, teaching experience, and concern for 
undergraduate learning were most important when evaluating programs 
totraincollegeteachers. 

The importance of knowing about some resources (Table B.3) also 
varied by program purpose; laboratory and computer facilities and 
external financial support are examples. But different program pur- 
poses did not appear to influence very greatly the importance ratings 
assigned to student characteristics (Table B.2) or the desirability of 
information about various operational aspects of programs (Table B.4). 
These results suggest that it may be possible to Vary information 
about a few selected characteristics according to the primary purpose 
of a program under study, rather than setting up different assessment 
procedures for each kind of program. 

In general, characteristics with a mean rating below 2.0 
(1 = "Essential;" 2 = "Important") for at least one type of program 
(last column in Appendix, a) were considered to have been endorsed 
by the panel and were included on the second questionnaire. In 
addition, a f ew charac t er is t ics were re-structured based on comments 
of respondents and apparent overlap. These criteria eliminated 5 of 
the 15 characteristics originally listed under "Faculty Members," 
8 of the 17 characteristics listed under "Students," two of the 
10 characteristics listed under "Resources," and 6 of the 21 char- 
acteristics listed under "Program Operation." In general, the Chirac 
teristics omitted from the second questionnaire are the ones toward 
the end of each table in Appendix B. 

Though one purpose of the first questionnaire was to eliminate 
some characteristics, the detailed results in Appendix B warrAt 
further attention along two lines: the differences In rated impor- 
tance depending upon the purposes of Ph.D. programs, a^ noted above, 
and the distribution of importance ratings assigned to each character 
istic by panel members. Though the mean ratings provide a convenient 
index for the identification of the most important characteristics, 
the frequency tabuIati(Mis of ratings give a better indication of the 
degree of consensus among graduate deans about the importance of 
each character is t ic . For example , Table B. I reports that more than 
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80 perc^nt*^ of the respondents agrteed that it is essential to know 
about the research activity and productivity of faqul'ty members who 
are training research scholars. But they did not agree to ^he 
same extent, about the importance of similar information concerning 
faculty members who are training professional practitioners. Also 
in" this table, there is disagreement over the importance. of knowing 
about faculty concern for student development and welfare, teaching 
effectiveness, involvement in program affairs, or group morale when 
making judgments concerning the quality of programs to train research 
scholars. Similar ^examples of apparent consensus or difference of 
opinion can be found in the tables on Students, Resources, and 
Program Operation. These different points of view among experts 
in graduate education have implications for the development of 
any systematic approach to the assessment of quality in Ph.D. 
programs of study. ^ 



Second Questionnaire: M ea^u^rjjig^-Jmp^ Characteristics 

Frequency tabulations and meafi ratings on the second question- 
naire are reported in detail in App^dix C. As vrith the importance 
ratings of characteristics, the mean ratings of measurement adequacy 
are useful as an index tc identrifv those measures generally con- 
sidered "very good" or "good" by the panel of graduate school deans, 
but tlie frequency ^is trj.bu t ions of ratings give a better indication 
about the dfgree of consensus among the deans. For instance, the 
first characteristic under Faculty (Table C.l) includes "appropriate- 
ness of specialty areas of training to purposes of the program, as 
evaluated by a visiting team of experts," which received a mean 
rating of 3.0 or a "good"^ indicator of the academic training of 
faculty. But this mean rating masks the fact that an equal number 
of deans rated the measure "very good" and "poor or inadequate." 
It was preferred by a few, but two other measures, received much 
higher preference endorsement. Also, information about the two 
more preferred measures is already available on more campuses. 
Therefore, though this measure received a "good" average rating, 
two others probably are potentially more useful. On the other hand, 
it was rated higher than three other possible measures on the list, 
though each of these measures was considered "very good" by a few 
graduate deans and two of the three were preferred measures by a 
few respondents. 



The rating scale on the first questionnaire (Appendix B) was 
l=Essential to 4=Not important. The rating scale on the second 
questionnaire (Appendix C) was l=inadequate to 4=very good. In 
the 'summary of the data (Appendix A) the rating scale on the first 
questionnaire has been reversed so that higher mean ratings always 
mean "more important" or "more adequate." 
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A similar comparative analysis c6uld be made among the measures 
suggested for each program characteristic listed in Appendix C, and 
pr9bably^ should be done by a group of everts in graduate education 
who could interpret more £ully the meaning of the results for the 
measurement of each characteristic. Such an extensive analysis is 
beyond the scope of this brief report, though it is recommended as 
a fruitful next step by an appropriate group. There are a few more 
general comments about the ratings of the measures, however, that do 
seem appropriate at this time. 

First, there is generally good agreement between high mean ratings 
on measurement adequacy and preference for a measure. Preferences 
sometimes scatter over several items, particularly if a number of 
possible measures were suggested for a given^pharacter istic , but 
generally 70 to 80 percent of the first pref e^t'ences were received by 
measures that also received a mean rating of 3.0 (''good") or above. 
Occasionally a measure was preferred by a number of deans even though 
it was not considered a particularly good index, such as a measure 
for characteristic 10 in Table C.l (.concern for undergraduate learning) 
that was preferred by 15 deans but had a mean rating of 2*86, but this 
is unusual. In general, the concurrence of mean ratings and preference 
choices is an encouraging indication that these measures are acceptable 
to graduate deans and therefore likely to be useful in the assessment 
of program quality., ;i 

Second, for most program characteristics there are one or two or 
three measures that were definitely rated higher and^ preferred more 
often by deans, while there are some measures that were definitely 
considered poor or inadequate by an appreciable number. Consensus 
was great enough so that, on most characteristics, a clear line can 
be drawn between the measures that are endorsed by the panel of deans 
and those that are conttoversial or inadequate. Therefore, though 
the list of items on th.e second questionnaire was very long and 
burdensome for respondents, thei]- ratings; clearly recommend the 
development of some assessment procedures rather than others. En- 
dorsed characteristics and measure^could provide the framework for 
a model to assess quality in Ph.D. programs that Would be appropriate 
for use at a variety of universities. These results also may be 
helpful to those who are attempting to choose among various possible 
ways to assess, Ph.D. programs in- specific settings.. 

Third, while some, of the standard input measures are endorsed 
as indexes for s^everal' of the program characteristics, self-report 
measures from program participants (program directors, faculty mem- 
bers, and students) and judgments bv both a visiting panel of experts 
and by program participants are rated highly as indekes for a number 
of characteristics. The endorsement of iudgments by a visiting panel 
of experts is consistent with tfh^ cuVrent practice at a number of 
universities to use outside teams as part of the evaluation process 




(McMichael, 1973) and yupports a recent recommendation to the Board 
of Regents concerning ^che evaluat ion of graduate education -in New York 
(Fleming, 1971) . The /support for ratings, jud^gments, and reports from 
faculty membeLr-s and Students w^s not so predictable, but , it Yfas even 
more important implications for the development of procedures for 
program assessment . These ratings suggest 'that some standard methods 
to colled the judMent's and opiniorfs of facuitv members and students 
o*n selected aspect^ of their .graduate programs would be welcomed and ' 
endorsed by many' graduate school d^ans. 



Fourth, thes^ data indicate^ that much of the more acceptable or 

lot De 



adequate and preferred information needed to mak^^^dgments about the 
quality of Ph . D . /programs is not currently availal^Ron university 
campuses. The "Availability" responses should not^e interpreted too 
literally, since comments on the second questionnaire suggest that 
somewhat differj&nt interpretations were given to the question. But 
it is probablyKaccurate to conclude that information is generally 
available for only about one out of five of the measures that received 
mean ratings of '3.0 or above. Standard procedures to collect some of 
the currently ainavailable informat ion shoulil be we 1 come d by the gradu — 
ate sthools as well as by other agencies interested in program evalua- 
tion. ' 



Summary List / of Measures for the Assessment of i:?uality in Ph^.D. Programs 

! ■ . 

Appendiix A summarizes the results of two questionnaires to a 
representative panel of graduate school deans by listing the character- 
istics of Ph.D. programs in the order of their rated importance for 
judgments about quality, and listing the most acceptable or adequate 
measures fax each characteristic in the order of their rated adequacy. 
Characteridcics ^nd measures are grouped under the same four headings 
used in Appendixes B and C: Facuitv Members, Students, Resources, and 
Program Operation. Items under each heading are from the more detailed 
reports (Appendixes 6 and C) but their orders have been changed in the 
summary likt to reflect their relative endorsement. Characteristics 
listed under each heading in the summary range from "essential" 
(ratings a'f 3.5 or above) to "important" (ratings 3.4 to 2.6). Most 
measures t|ave a mean adequacy rating of 3.0 or above ("good" to "very 
good") , tljough occasionally a measure with a rating slightly below 3.0 
that is preferred by a ^number of panel member^s is included in the list. 
Notes preceding the sunjmary list define the terms and symbols used in 
the tables . 



This 



listing of program characteristics and measures is not 
proposed ks a definitive view of quality , in Ph.D. programs of study, 
but rathej: as a first winnowing of the multitude of characteristics 
and measures that might be considered in the assessment of such a 



complex and varied set of educational efforts. The list as it 
stands now would profit from further review by experts in the field 
of^ graduate education, and probably could he shortened and simplified 
somewhat. Howevei', even in its present form it represents a better 
picture of Ph.D. program assessment procedures that would be accept?^ 
able to a cross-section of graduate school deans than has been' 
available so far. As such, it should be useful to the graduate 
education conmiunity, as well as to others who are interested in 
furthering the. systematic development of procedures to assess 
quality in graduate education. 

♦ - 

Conclusions and Recoimnendations 



This report presents the opinions of a sample of graduate 
school deans concerning the importance of various program character- . 
istics to judgments about quality and the adequacy or acceptability 
of. several ways to measure each characteristic. In addition, program 
cbTaracteristics are evaluated in relation to the primary goals of 
preparing research scholars, college teachers, or professional 
practitioners, and some estimate is made about the present avail- 
ability of information suggested by the measures. All ratings 
c®r»c^rn Ph.D. programs of study, and emphasize dimensions of quality 
that are commo*n to many fields of study. 

The mps't important characteristics and -the most acceptable 
measures are summarized as a first step toward the development of 
a model to assess quality in Ph.D. programs of study. This listing 
can provide guidelines for efforts at individual universities to 
establish appropriate evaluation procedures, and also provides a 
framework for more detailed work on the development of sys^tematic 
procedures for the assessment of quality in doctoral education. 
Several next steps are possible; the foll6wing three recommendations 
are suggested as ways to begin to refine and build on these data. 

First, more research should be done on differing points of view 
about the importance of program characteristics and the adequacy of 
measures. There may be identifiable subgroups of opinion within this 
sample of^graduate school deans; more detailed analysis of their 
responses would be needed to determine whether their opinions are 
moderated by their oum academic training and fields of specialization, 
or type of employing university, or other conditions. In addition, 
other publics, such as faculty members in selec t ed . D . programs, 
students, accrediting* groups, or legislators, might have similar or 
different opinions about the relationship between various program 
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characteristics or measures and judgments about quality. Consensus 
across groups as well as among graduate school deans could be 
determined through "school s of t bought " analytic procedures sug- 
gestefi in the first stage of the earlier proposal for research on 
quality in doctoral programs (Clark Hartnett, 1972). 

Second, discussion of the results by a group of 'experts in 
graduate education would be useful. What difficulties may be., 
encountered in attempting to collect the preferred but unavailable 
kinds of information? How much weight should be given to the 
purposes of ^ program when deciding upon characteristics to be 
measured, given the importance ratings for different program pury 
poses? What interpretation should be given to the -dif f erences in 
points of view about the importance of characteristics and the 
acceptability of measures within this sample of experts in graduate 
education? How can a research and development project minimize (or 
ut il ize) these differences when developing some systematic procedures 
for program assessment? A special group might be gathered to discuss 
the results, orr such discussions might be initiated at scheduled 
professional or association meetings. 

Third, the summary of important characteristics and acceptable 
measures could be used as a framework for a model to assess quality 
in Ph.D. programs which could be tested empirically (second stage 
of the Clark hnd Hartnett proposal). Research along these lines 
would develop procedures to collect preferred information and then 
use these procedures to obtain data from selected doctoral programs 
in order to .examine their reliability and validity as well as 
evaluate the relationships among variables selected for study. 
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Appendix A j 
A Suminary List of Measures for the Assessment ^ 
of Quality in Ph.D. Programs 

Notes ' ' \ J 

1. Importance ratings of program characteristics : 

A = Essential to have information afc^ut tliis characteristic in 
order to judge quality 

3 = Important to have this; information , 

2 = Perhaps useful; not particularly important 

1 = Not important to judgments about quality 

(This scale reverses the one used in Appendix B t<) tabulate 
results of the first questionnaire; in order to obtain 
greater ease of interpretation.) 

2. Mean importance ratings in this summary are the. inverse of the 
grand mean of the ratings for each of the three program purposes 
reported in the last column of Appendix B. A specific assess- 
ment plan should consider these more specific ratings depending 
upon the primary purpose of the programs, to be studied. 

3. Adequacy ratings of measures; 

4 = Very good indicator of the quality of given characteristic 

3 = Good indicator ^ 

2 = Poor but useable indicator 
1 = Inadequate indicator 

4. The "percent preferred" column represents tjhe percent of 43 
respondents who selected this measure as thft^r first-choice 
or preferred measure for a giv^n characteristic. 

5. The "percent available" column represents the percent of 43 
respondents who indicated that the ' inforraiat ion could be supplied 
from records without new data collection. Responses generally 
rdn 37 to 40 out' of 43; therefore, 10 to 15 percent of the total 
often are non-respondents. The number of deans who say the 
availability of information varies from department to depart- 
ment, in addition to being generally available, can be 
determined from Appendix C. 

Note especially that the characteristics in each section of this 
list are also listed in comparable sections of Appendixes B and 
C, and the measures in each section are also listed in comparable 
sections of Appendix C, but the order of arrangement is dif^ferent 
to reflect the resul ts of import ance and adequacy rat ing^ on the 
two questionnaires . 
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Characteristic 



Table kd 

^ FACl^Kharacteristics and Measures 




. I 

^ I 



Hean * 
Importance • , ' 

3.57 A. A(||deniic training of faculty 

1. Percent with Ph.D.'s or equivaleijt degrees 

\ 

\ 

1. Percent of Ph.D.'s from "top" programs in each field, 
as defined by a reputational survey 

, 3. Appropriateness of specialty areas of training to purposes 
of the program, as evaluated by a visiting team of experts 

f 

^3,41 B. Research (or other scholarly or creative) activity 

1. Average number research proposals funded in the last three 
years per FTE faculty 

2. Average number invited presentations of research results 
in the past year (visiting lectures, colloquia, workshops, 
professional meetings, etc.) per FTE faculty , 

<s^^ 3. -Percent of faculty actively involved in the publication of 
research results (journal editor, editorial board members, 
referee for submitted articles, etc.) 

4. Average number research projects in progress per FTE 
faculty ' ^ 

3,20 C. Research (or o^her scholarly or creative) productivity 

1. Weighted average number publications (giving progressively 
morf weight do refereed journals, single authorship of 
articles, seMor authors^iip of books, etc.) in the past 
three years/(sub. ho. patents or artistic performances or 
products where appropriate) 



2'^ 

ERIC 



2. Average number journal articles published in th^ last three 
years, per FTE faculty 



Heasyres 

Mean Peycent, Percent 
Ade£uac2 Preferred \Available 



3.37 



3.26 



3.02 



3.51 



33% ioo;s 



3.33 , 42 



3.00 09 



3.40 30 



3.33 16 



14 



■26 



3.74 ■ 77 



09 



65 



12 



09 



37 



49 



21 



28 



67 



Faculty (cont . ) ' • 
/ 'Characteristic .. ' ^ , Heasures 

( Mean ^ ' .Mean Perceflt Percent 

' ^"'P^'^^^"'^^ ' , • Adequacy Preferred Available 



I 



3.17 D, Teaching effectiveness ' 

1, Avg. ratings by students on ^ course and teacher evaluation form 3.'10 VI lit 

h Presence of Innovative teachinf; procedures and/or outstanding 

Instructional materials, as jud^ned by visiting panel of experts IK) 16 07 

1 

1.11 L Concern for stud'ent development and welfare 

i. Averty^e student (or former s-tudpnt) rati.ng on a scale measuring 
laculty concern lor student dei/elopment and welfare, accessi- 
bility, advising skill, etc. ' 1,21 % 07 



2.94 / F, Involvement In program af^irs 



; « ■ 1. Faculty degree of agreement with m\ conrnitmejit the 
I / purposes and goals of the prograi'i » , l,2f) ' l\l 05 

1. Faculty satisfaction with influence on important de. isions 

concerning the program .. \ 1.21 ]1 07 

, 2.93 , G, Teaching experience • . . ' ' 

1. Avg. no. of years of teaching experience at the col lege 

' university level In appropriate area of specialty * 12 ' 70 

2M 11, Honors and awards; national professional recognition 

I 

1. Average mimher proftv^sloiial hmr^ m\ awards pof FTE 

faculty over the past three vears^ l.T] 40 35 

Average reputational rating of the faeiilty among eolleagues 

(i^f, A(-F ratings) ^ ']M , .;30 37 

ERIC 



Characteristic 



Faculty (cont.) 



Mean 



iaportance 



I 
I 

9 



Measures 

Mean Percent Percent 
Adequacy Preferred Available 



1 



2.78. 1.. Group morale or esprit , 



1. Avg..'' faculty-reported satisfaction with program leadership, 
' enthusiasm for the program, loydty, involvement, etc. 



3.45 



2. Relative vitality vs. malaise among the faculty, as rated 



5U 



by a visiting panel or experts 


3.23 


14 


09 


2.55 J, , Concern for undergraduate learning 








. 1. Quality of a program to train and supervise teaching fellows, 
as rated by: , 








^ a) students 


3.12 


14 


■ 07 


b) faculty members 


2.95 


02 


02 


c) visiting panel of experts t ' 


2.93 


07 


,05 


2. Faculty-reported sensitivity to the interests, needs, and 
aspirations of. undergraduates 


2.86 


35 


02 


t ♦ 

3. Percent of graduate faculty who teach at least one under- 
graduate course per year , 


2.86 


14 


67 



27 



28 : 



Table A. 2 

STUDENT Characteristflcs and Measures 



Characteristic 



Mean 
tffpoftance 

3.61 A. General academic ability of students entering the pro^ran 
*1. Avg. score on graduate aptitude test (GRE, MAT, etc.) 
2. Avg. undergraduate GPA . 



Measures 

Mean Percent Percent 
Adequacy Preferred Available 



I 
I 



3.61 



ERIC 



4. Percent of enrolled students with national fellowships 
(NSF, Woodrow Wilson, etc.) 



B. Achievements, knowledge, and/or skills of students at time of » 
completion of degree \ 

1. Excellence or uniqueness of dissertations and theses iii the 
last three years, as evaluated by external judges 

2. Percent of graduates in the last three years who obtained 
employment directly related to their field of specialization 

3. Percent of graduates in the last three years who published 
something prior to the degree 



3.38 C. Professional accomplishments of 'graduates 

1. Percent of graduates occupying positions of leadership and 
influence' in the field, as judged by outside experts- 

2. Avg. no. publications by those who graduated in the last 
five years 

3. Avg. rating by employers on knowledge, skill, and performance 
of recent graduates 



3.42 



3.37 



3. Avg. score on appropriate GRE Advanced , Test (or Other ach./fest) ' 3.09 



3.07 



3.47 



3.33 



3.24 



3.37 



3.30 



3.14 



4, Percent of graduates currently employed by doctorate-awarding 
universities 3.09 



111 



35 



07 



09 



26 



26 



23 



23 



21 



12 



12 



79 



60 



65 



05 



47 



28 



02 



09 



32 



mim (cont.) 

Characteristic . • Measures ^ 

Mean . . ' . ' Mean - Percent ' Percent 

importance i Adequacy Preferred Available 

3.37 D, Gains in student knowledge and/or skills during their degree prograi 

1. Avg. evaluation by faculty advisers and by assistantshlp/intern- 

ship supervisors of the "growth" observed in students over a two ' ' 

year period • . 3.12 402 192 

3.15 E. Career interests of students , ^ 

V 

1. Congruence of student career interests with program purposes ■ • . ^ 
and emphases, as judged by enrolled students ' 3.10 44 07 

2, Self-reported career plans of degree recipients 2.93 14 '16 
g 2.91 F. Student perceptions of program quality 



I 



1. Avg, alumni (1-5 years) rating of experiences in the program ' 3.58 72 '02 

, ' 2. Avg. student rating of the value of .specified academic 

experiences ip the program , ' 2.98 12 07 

r 

2:89 ' G. Satisfaction with various aspects of the degree program 

1. Percent of students who would recomraend the program to others 

with similar interests and abilities 3.40 51 05 

2. Avg, student-rated satisfaction with specified academic and 

non-academic aspects of the program ' 3.02 26-09 



ERIC 
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STUDENTS (cont.) 



Characteristic 

' Mean ' 
imporc^nce 

2.81 H. Group tnorale or esprit 



, 1. Student sense of comunlty, feeling of shared Interests and 
Involvement In worthwhile activities, as rated by students 

2. Avg. student satisfaction with rate of "acad^lc progress, 

■V 

3. Avg. student-reported sc6r6 on a scale measuring satisfaction 
with program . 



.Measures 

Mean Percent Percent 
Adeqmcy ' Preferred Available 



3.30 , m ' m 



2.88 12 



2.86 26 



07 



07 



I 

u 

H 

I 



3^ 
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. , V Table A.3 

RBSOURCES and Measures 

Characteristic 

Mean * • 

Importance 

3.60 A. University financial support for the program 

6 

1. ^Judgments about the adequacy of university support relative 
to program purposes by a visiting team of experts 



i 



Measures 



Mean Percent Percent 
Adequacy Preferred Available 



I 
I 



3.10 



m 



m 



2. Education and general ejcpense budget per FTE student 


3.00 5, 


14 . 


70 


J. Ratio of program budget allocation to total university 
allocation for Ph.D. study 


2.83 


23 


50 


3.60 B. Library 








1. Adequjjicy of relevant holdingSvas judged by: 








a) visiting team of experts 


3.45 


44 


21 


b) faculty members 

t ' 


3.43 


26 '. 


28 


< 

c) program chairman 


3.07 


0 


23 


3.47 C.^ Laboratory equipment and facilities (including facilities for the 
creative arts) ^ 








1. Presence of laboratory/creative equipment considered essential 
or important by experts in the field 


.3.56 


42 




j. Adequacy of laboratory equipment and facilities as r^ted by; 








• »a) visiting team of experts 


3.53 


19 


16 


, b) faculty members 


3.40 


28 


23 



o 35 
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RESOURCES (cont.) 

Char^cterlGtic 
Mean • 

Importance ' ' 

3.18 D. External financial support for the program , 

. Outside fund8 as a percent of total prograin budget ' • 

. Avg. dollar income per PTE faculty from outsidp sources over 
the past three years 

. Dollar amount of federal research project grants and contracts 
over the past three years ^ 



Measurefi 

•{If 

' Mean Percent' Percent i 
■ idequacy Preferred Available 



I 
I 



3.13 E. 



3.11 F. 



37 



urposes and strengths of parent institution 

. Reputation of university as rated by knowledgeable educators' ' 

. Congruence of program and university purposes as judged by 
visiting panel of experts ' 

. Level of support for the program from central university 
administrators, as judged by visiting panel of experts 

inancial -support for students 

. ^g. dollars for all forms of financial assistance per 
m student 

. Percent of students receiving any form of financial assistance 

. Percent of students «ho hold fellowships or grants from 
external sources ' i. 



3.28 



3.16 



3.49' 



3.07 



3..45 
3.1^ 

2.98 



;2 



33 



3.19 / 14 



12 



49 

■fi23 

12 



65t 



60 



77 



21 



16 



12 



67 



'tev 72 



77 



38 
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'Ix Characteristic 

Haan 
tmoortaiice 

3.07 G. Computer {aclllties 



afeSOURCES (cont.) 



I 
I 



5.75 



1. Adequacy of computer facilities for needs of the program as 
.. rated by: ' 

a) visiting team of experJS^^ 

b) faculty members > , 

* f 

c) progiam chairman 

d) students , 

■ ■* ■ , ' 
H. Classrooin and office space for the program 

1. Adequacy of classroom and office space as rated by: 
' • a) visiting team of experts 

b) faculty members 

2. Avg. square feet, of office and research space per FTE faculty 



Measures 

Im Percent Percent 
Ade( |uacy Preferred Available 



3.44 


332 


14Z 


\ 17 


91 




U4 


05 


■ 33 


3.12 


::07 


. 23 * 


t 






3.05 


32 


14 


2.91 


19 .' 


19 


2.91- 


14 • 


07 



33 



40 



I 

M 
I 



Table A.^ 

OPERATIONAL CharacListics and Measures 



Characteristic 



Kean 
inportance 

3.49 A. Purposes of the program 

1. Clarity of program purposes and plans, as judged by: 



Measures 

Mean Percent Percent 
Adequacy Preferred Available 



a) visiting panel of experts 


3.28 


231 


07; 


b) recent graduates 


3.26 


14 


02 


c) faculty members 


' 3.19 


21 


16 


! 

d) program chairman and/olj: graduate dean 


3.U 


09 


23 


2. Relative emphasis on research, teaching, and service, 

as indeed hv vi<?iM'no hpam ovnorfc 


J.UD 




14 


Course and program offerings 








1. Courses and other available experiences appropriate to 
purposes of the program and specialty training of the 
faculty, as judged by: 








a) visiting panel of experts 


3.56 


'44 


09 


b) faculty members 


3.26 ^ 


23 


19 


c) students 


3.05 ' 


0 


09 


2. Relative need for curriculum review and updating as 
judged by a visiting panel of experts 


3.32 


12 


09 



ERIC 
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OPERATIONS (cont.) 



Characteristic 



00 

I 



43 



Mean' 
importance 

' 3.43 C. Admissions policies 

1. Judgment cif whether admissions standards should 
be higher or lower, as rated by: < 

a) visiting panel of experts 

b) faculty members 

2. Percent 'of qualified applicants who are admitted, 
as reported' by the admission comniittee 

3. Relative selectivity of graduate and undergraduate 
students, as reported by the admissions committee 

i Clatity of admissions procedures and standards, 
as judged by visiting panel of experts 

(not rated) D. Provision for the welfare of faculty members 

\ V 

1. Median salary by rank 

2. Extent of department and university support for the 
principle of academic freedom, as Judged by outside experts 

s 3. Ex(;ellence of fringe benei|its package, as judged by 
outside experts 

I 

4. _ Faculty satisfaction with freedom to plan courses and 
conduct research without internal or external interference 



Measures 

Mean Percent ' Percent 
Adequacy Preferred Available 



3.39 
3.05 

3.09 

3.07 

3.00 



3.26 



3.23 



3.10 



3.07 



09 



21 



23 



21 



12 



02 



19 



1231 
21 

67 

53 

12 



66 



14 



16 



09 



(continued) 
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OPERATIONS (cont.) 



Characteristic 



Mean 
Importance 



5. Provision for assistance to new and young faculty, 
as judged by; 



3.U E. Provision for the evaluation of student progress 

1. Clarity of requirements and standards for progression 
from entrance to candidacy with stated times for 
review and evaluation, as judged by; 



3.37 F. Program leadership and decision-making 

1. Quality of leadership provided by the program 
chairman as judged by; 



Measures 

Mean 'Percent Percent 
Adequacy Preferred Available 



a) visting panel of experts 


*3.07 


075; 


072 


b) faculty members 


3.00 ' 


•0/ 


, U 


6. Average teaching load 


3.02 


or 


60 



a) 


visiting panel of experts 


3.16 


26 


12 


b) 


program chairman 


■ 3.16 


21 


30 


c)' 


students 


3.U 


21 


16 



a) 


visiting panel of experts 




3.44 


21 


09 


b) 


graduate dean 




3.40 


19 


37 


c) 


faculty members 




3..37 


33 


16 
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OPERATIONS (cont.) 



Characteristic 



, Mean ' , ' 
importance ' 

3.31 G.' Job piacement of graduates 



I 

0 

I 



i 



Measures 

Mean Percent Percent 
uacy Preferred Available 



1. Percent of degree recipients in last three years who were 
placed in positions directly relevant to their graduate 
education 


3.56 


m 


421! 


2. Satisfaction with program efforts to place graduates, as 
/ judged by students 

■I. . . ' • 


3.26 


^ 14 


. 12 


3. , Percent of recent graduates who joined faculties at major 
doctorate-producing universities 


3.07 


07 


51 


4., Profile of jobs taken by former students and graduates 
in the last three years 


3.00 


19 


14 


3.30 H. Provision for the advisement of students 








1. Quality of the advising system, ^as. rated by: 








a) recent graduates, : 


. 3.42 


40 


02 


b) students 


2 42 

> 






c) visiting panel of experts 


3.02 


05 


07 


3.29 I. Student-faculty interaction 

\ 








/ 

1. Student-reported satisfaction with opportunity to work 
closely with at least one member of the faculty 


3.42 


46 


05 


\ 
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OPERATIONS (cont.) 



Characteristic 



Mean 
importance 



\ 



3.17 J. Internships or other opportunities for relevant student 
experiences 

f , 1. Value of the internship or assist/ntship as rated by 
students who have completed the experience 

\ 

2. Percent. of students who hold a research assistantship 
in the course of their attendance 



3. Percent of students who" hold a teaching assistantship 
In the course of their attendance 



^ 4. Percent of students who hold an apprenticeship. or other 

T preprofessional work assignment the course of their 

attendance 



3.16 K. Degree requirements 

■ ■ ' . ■ t 

1. Flexibility of program requirements sufficient to meet 
individual student needs, as judged by: " 

a) visiting panel of experts • ' 

b) students ^ 

c) faculty members 



2. Clarity of specified competencies and qualities expected 
of graduates preparing to be researchers, teachers, or 
professional practitioners, as evaluated by; 



f j( ^ a) visiting panel of experts 

b) faculty members 
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OPERATIONS (cont.) 



Characteristic 
Mean 



Measures 
Mean Percent Percent , 



importance 






Adequacy 


Preferred 


Available 


,3.07 


L. 


/ 

Enrichment with visiting lecturers, coUoquia, kc, 








• 




1. No, visiting lecturers, colloquia, etc., scheduled in 
the past six months 




463: 


49Z . 


\ 




2. Satisfaction with enrichnent efforts of the program 
as rated by: ' 












a) visiting panel of experts 


3.13 


19 


05 


1 












1 




c) faculty members J 


3.12 


05 


14 


2.94 


M. 


Relationships with cognate programs ^ ' 


• 










1. Percent of progra students enrolled for one of more 
courses in another department * 


3.14 


26Z 


51Z 






2. Relationships and intltchanges with cognate programs 
as rated by: 






• 






a) program chairman and/or graduate dean 


3.10 


16 


23 






b) faculty members \^ 


3.05 


14 


19 


i 




c) students 


3.05 


07 


14 


51 




d) chairmen of cognate departments 


3.05 


'07 


21 
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OPERATIONS (cont.) 



Characteristic 



Measures 

Mean Percent 
Ade<iuacy Preferred 



percent 
Available 



N. Efficiency of degree production 







1. Percent of those who enroll vho earn the Ph.D.- 


3.A5 


422 


72S: 






2. Estimated total avg. time required to complete the degree 


3.14 


09 


58 






3. Ratio of doctorates awarded to no. of graduate faculty and 
to no. of enrolled students 


3.02 


23 


'65 


86 


0. 


Plans for the future of the program 

1. Appropriateness and detail of middle-range and long-range 
plans for the program, as evaluated by; 












a) visiting panel of ^experts ^ 


3.37 


46- 


12 






b) faculty members 


3.07 


12- 


19 






c) program chairman aiid/or graduate dean 


■ 3.02 ■ • 


15 


30 


83 


P. 


Size of the program 


\ 

1 










.1. 1^0. of enrolled students and first-year students 

in each of the past four years, by full-time and part-time 


3.31 


. <*(> 

v", 

28 


84 






2. Student/faculty ratio 


3,07 


84 






' ■ \ 

3. No. of graduate faculty members in each of the past four 
years, by full-time and part-tlipe 


3.00 


05 


77, 



r 
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* i Appendix B 

The Importance of Program Characteristics 
for Judgments about Quality 

Notes ' 

1« Instructions to raters were as follows: 

For each of three types of doctoral programs — i.e., for those 
emphasizing the preparation of researchers, teachers, or practitioners-- 
indicate how Important it would be for you to have information about 
each listed program characteristic in order for you to make a. judgment 
about a program's quality. Make your rating by writing a number in 
each .column for each item, using the following rating code: 

1. Essential ~ must have information about this in order to 
judge quality; this element of a program/ is critical to the 
achievement of excellence in doctoral education 

2. Important — desirable to have this information if at all 
possible ^ 

3. Perhaps useful; nice to know if easily available but not 
particularly important 

Not important to judgments about quality 

2. Tables 1 through 4 report^.tiie ratings assigned respectively to charac- 
.teristics of faculty members, sti^dents, program resources, and program 

operation. Within each set, the characteristics are listed in the order 
of their importance to quality judgments about programs to train research - 
scholars. ■ 

3. Note that, in these tables, a higher mean rating is the lower number. For - 
instance, these graduate deans say that it is more important' to know about 
the research activity of a faculty member in a program to train researchers 
(mean rating of 1.08) than in programs to train teachers or practitioners 
(mean ratihgs of 1.71 and ^1*98 respectively.) 

4. Tabulated ratings of^ll program characteristics listed in the' first 
"questionnaire arG'^Tncluded in these tables. 





Table B.l 






JLmportant Infomlatlon for Judgmants about the 
Educational Quality^ of Ph.D. Programs: 






CHARACTERISTICS OF FACULTY MEMBERS 






Number of graduate deans who rated 
Information about the characteristic:* 




Characteristic 


Perhaps useful; 

In programs Essential Important not Important 
% to train: (1) (2) (3 and 4) 


• Mean 
rating 



1. 


Research (or other 


Researchers 


46 




scholarly or creative) 


Teachers 


17 




act ivltv 




IS 


2. 


Research (o r at her 


Researchers 


Al 




scholerly or creative) 


Teachers 


14 




p !• uuui. L X V X luy |# 


rracci t loners 


1 A 


3* 


Academic training 


Researchers 


35 






Teachers 


29 






rract it loners 




4. 


Honors and awards; 


Researchers 


18 




national professional 


Teachers 


1 




recogni c ion 


rract 1 tioners 


A 


5. 


Concern for student 


Researchers 


14 




development and 


Teachers 


23 






rract 1 tioners 


1 / 


o • 


oA/^n'4nO r% C f A f> ^ A n. n r% ■ 

icaL.nxii{^ c I cc u 1 venes 8 


Researchers 


1 9 
1 Z 






Teachers 


39 






Practitioners 


11 


7. 


Involvement in program 


Researchers 


12 




affairs 


Teachers 


11 






Practitioners 


10 


8. 


Variety of specialties 


Researchers 


9 




among members^yot the 


Teachers 


8 




faculty \ 


Practitioners 


4 


9. 


Teaching experience 


Researchers 


7 






Teachers 


30 


* 




Practitioners 


4 


10. 


Group morale or 


Researchers 


10 




esprit 


Teachers 


8 






Pracjc it toners 


9 



4 

29 
20 

7 
20 
11 

14 
19 
23 

25 
35 
25 

22 
21 
15 

26 
10 
23 

» 

23 
27 
21 

29 
31 
34 

22 
14 
21 

26 [ 
23 



3 
14 

1 
14 
23 

1 
1 

6 
12 
19 

14 
5 
17 

12 

15 

13 
9 
16 

12 
10 
11 

21 
5 
24 

23 
15 
17 



1.08 
1.71' 
1.98 

1.16 
2.00 
2.25 

1.32 
1.43 
1.55 

1.76 
2.30 
2.31 

' .j . 

2.02 
1.63 
2 .02 

2.08 
1.20 
2.20 

2.08 
1.96 
2.15 

2.08 
2.04 
2.16 

2.2Z 
1.49 
2.^9 

2.32 
2.14 
2.20 



*Tqtal N>>50. Rows may not add to 50 because respondei^ts sometimes omitted an 
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CHARACTERISTICS OP FACULTY MEMBER* 



Number of graduate deans who rated 
Infonaatlon about the charactarlatlci* 



Characteristic 


In prograna* 

to train: 


Easantial 

(1) 


Important 

(2) 


Perhaps useful; ' 
not Important 
(3 and 4) 


Mean 
ratln{ 




• 






« ■ ' 




11 • Turnover 


Researchers 


8 


19 


18 


2,33 




Teachers 


5 


16 


23 


2.58 




^ L oCL XL loners 


5 


1 s 


^ J 


2 . 56 


12. Concern for under- 


Researchers 


4 


17 


29 , 


2.72 


graduate learning 


Teachers 


22 




8 


1.78 




£^ L L X L xuner s 


3 


12 




Z . o4 ^ 


13. Non-unlverslty 


Researchers 


1 


11 


37 


2.98 


activity 


Teachers 


1 


16 


31, 


2.73 




Practitioners 


6 


18 


24 


2.50 


14. Involvement in univer- 


Researchers 




9 


41 


3.10 


' ' slty affairs 


Teachers 


3 






2.65 


' i 


Practitioners 




5 




3.18, 


15. Concern for meeting 


Researchers 


3 


6 


41 


3.14 


local and community 


Teachers 


5 


17 


27 


2.71 


needs 


Practitioners 


11 


17 


21 


2.27 



t 
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Table B. 2 

important Information for Judgnanta about the 
Educational Quality of Ph.D. Programa: 

CHARACTERISTICS OF STUDENTS 



Characteristic 



In programs 
to train: 



Number of graduate deans who rated 
Information about Che chatactetlstlc;* 

Perhaps useful; 
Essential Important not Important 

(1); (2) (3. and 4) 



Mean 
rating 



1. General academic ability 
of atijdents entering 
progr^ 

2. Achievements, knowl- 
edge and/or skills of 
students at time of 
completion of degree 

3. Professional accom- 
plishments of 
graduates 

4. Gains in studept knowl- 
edge and/or skills 
during their degree 
program 

5. Job placement of 
gtaduates 



6. Carreer interests of 
students 



Student perceptions of 
program quality 



Researchers 37 

Teachers 29 

Practitioners' 27 

Researchers ' 34 

Teachers 29 

Practitioners 31 



Researchers 26 

Teacljers 19 

Practitioners 21 

Researchers 24 

Teachers 22 

Practitioners 25 



Researchers 23 

Teachers 18 

Practitioners 18/ 

Researchers 17 

Teachers .16 

Practitioners 15 

Researchers 4 

Teachers 4 

Practitioners 4 



13 
19 
)20 

14 
16 
16 



23 
27 
22 

20 
23 
19 



23 
26 
27 

23 
25 
26 

36 

37 

38 



1.26 



1 

2 

1 

3 
1 



1 

3 
6 

5 
3 
5 



4 
5 
4 

10 
8 
8 

'8 
7 



1.43 
1.49 

1.33 
1.46 
1.38 



1.50 
1.67 
1.69 

1.61 
1.60 
1.67 



1.62 
1.73 
1.71 

1.66 

-%%l-84 
1.86 

2.12 
2.08 
2.06 



8. Satisfaction with 
various aspects of the 
degree program 

9. Group morale or esprit 



Researchers 5 

Teachers 6 

Practitioners 3 

Researcher"? 8 

Teachers H 

Practitioners 8 



29 
35 
32 

22 
24 
25 



16 

8- 
12 

20 
14 
16 



2.24 
2V04 
2.14 

2.30^ 
2.08 
2.20 



*Total N-50. Rows may not add to 50 because respondents sometimes omitted an item. 
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CHARACTERISTICS OP STUDEMTS 



Number of graduate daana who rated 
lnf6rmatlon about the characteristic:* 



Characteristics 


pLui^LcunB 

to train: 


Ct9 a en L X ax 

(1) 


iMkMA M«« ^ 

imporcanc 

(2) ' 


rernapa uaezuXf 
noc mporcanc 

(3 and 4);; 


nean 
rating 


10. 


Involvement In profes- 


Researchers 


5 


27 


17 


■ - 
2.37 




sional affairs 


Teachers 


1 * 


22 


25 


2.56 








4 


^7 




9 97 


11. 


Attrition rate 


Researchers 


3 


25 


20 


2.38 






Teachers 


3 


24 


20 


2.38 








3 


24 


20 


4t • JO 


12. 


Baccalaureate origins 


Researchers 


5 


24 


20 


2.41 






Teachers 


4 


18 


27 


2.61 






Prac tl tloners 


4 


' 20 


25 


2 S7 


13. 


Enrollment status of 


Researchers 




18 


19 


2.62 




students 


Teachers 


1 


14 


22 


2.73 






Practl tloners 


1 


17 


19 


2 68 


14. 


Involvement In program 


Researchers 




17 


30 


2.79 




affairs • ^ 


Teachers ^ 


4 


19 


23 


2.46 






Practitioners 




23 


24 


2.6!2 




Nonacademlc needs and 


Researchers 


3 


12 


34 


2.88 




concerns 


Teachers 


1 


19 


28 


2.73 






Practitioners 


1 


16 


30 


2.81 


16. 


Geographical origins 


Researchers 


1 


6' 


43 


3.32^ 




of students ' 


Teachers 






i 43 


3.35 






Practitioners 




7 


42 


3.67 


17. 


Involvement In Insti- 


Researchers 




4 


46 


3.32 




tutional affairs 


Teachers 


2 , 


11 


36 


2.92 






Practitioners 


1 


6 


42 


3.14 



4 
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Table B.3 



Important Infoniatlon for Judgmenta about the 
Educational Quality of Ph.D. Programa: 

CHARACTERISTICS OP RESOURBES 



Characteristics 



Number of graduate deans who rated 
Information about the charac teristic:* 

i Perhaps useful; 

In programs Essential Important not Important 
to train: (1) (2) (3 and 4) 



-1. University financial 
support for the 
program 


Researchers 

Teachers, 

Practitioners 


37 

2 7 , 
24 


12 
21 
23 


1 


2. Library 


Researchers 

Teachers 

Practitioners 


37 
29 
25 


11 
19 
20 


1 

3 


3. Laboratory equipment and 
facilities (Including 
facilities for the 
creative arts) 


Researchers 

Teachers 

Practitioners 


37 
20 
23 


11 
21 
21 


1 

7 
4 


4. Computer facilities 


Researchers 

Te^achers 

Practitioners 


27 
9 

11 


' 20 
18 
24 


1 

« 21 

12 


5. External financial 
support for the 
^ogram 


Researchers 
Teachers 
Practitioners ^ 


28 
10 
14 


16 
26 
26 


5 
12 
8 


6, Financial support 


Researchers 
Practitioners 


20 
14 
12 


22 
21 
18 


4 
10 
15 


7. Purposes and 
strengths of 
' parent institution 


Researchers 
Teachers 
Practitioners ■» 


12 
11 
10 


32 
33 
28 


6 
5 
11 


8. Classroom and 
office space 


Researchers 

Teachers 

Practitioners 


5 
4 
4 


26 V 

34. 

28 


19 
11 
17 


9. Availability/utiliza- 
tion of noncampus 
redourqes 


Researchers 

Teachers 

Practitioners 


4 

3 


^ 11 
14 
21 


34 
34 
24 


10. Facilities for leisure 
time and sports 
activities 


Researchers 

Teachers 

Practitioners 


1 


5 
5 

4 


44 
44 
45 



Mean 
rating 



1.24 
1.44 
1,5? 

1.27 
1.40 
1.54 

1.27 
1.73 
1.60 



1.49 
2.25 
2.04 

1.53 
2.04 
1.88 

1.65 
1.91- 
2.11 

1.90 
1.88 
2.04 

2.32 
2.14 
2.29 

2.73 
2.79 
2.46 

3.28 
3.29 
3.35 



*Total N"50. Rows may not add to 50 because respondants sometimes omitted an 
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tttportanf: InfopiMtlon for JudffBMts about the 
Educatldnsl QuAlity of/Ph.D. Prograns : 

CHASACrmSTICS OP PBDGRAM OPERATiOR 





i — 




(Nuaber of graduate deans who rated 
l4iforaat Ion about the characteristic:* 


. ; "I 


Characteristics 


In programsi.^ 
to train r . ' 


i 

Essential 

' (1) 


Important 


rerhaps usetui ; 
not.i^ortant' 
(3 and 4) 


• •' . '^■ 

MeanX 
rutlrfg 


1. 


rurpoBes or cne program 


i ' * 

Researchers 

Teachers 

Practitioners 


'. 30 
26 


16 
19 ' 
19 


3 •.: 
2 


4.45 ' 
lv55 - 
1.52 . 


2 


Aunissions policies 


Researchers 

Teachers 

Practitioners 


26 
23 
22 


20 

2^ 
' . 22 


3'. 

2 

4 


1.53 
1.56 
1.63 




* 'uvxBioii lor Cne 
evaluation of 
student progress 


Researchers 

Teachers 

Practitioners 


20 
20 
20 


28 
28 ' 
26 


2 


1.57 
1.58 
1.63 




v^ourse ana progrsn 
offerings 


Researchers 

Teachers 

Practitioners 


24 
2.4 
25 


23 
24 
^ 21 


3 
1 
3 


1.58 
1.53 
1.55 




rrogram leadership 


Researchers 

Teachers 

Practitioners 


22 
22 
21 


25 
24 
24 


3 
3 
4 


1.62 
1.61 
1.65 


6. 


Provision for 
advisement of 
students 


Researchers 

Teachers 

Practltioness 


20 
18 
19 


25 
25 


5 
3 
5 


1.70 
1.69 
1.71 


7. 


Student-faculty 
Interaction 


Researchisrs 

Teachers 

Practltlbners 


21 
21 
19 


22 
24 
23 


T • ,K 

4 

7 


1.72 
1.65, 


8. 


Degree requirements 


Researchers 
Teachers , 
Practitioners 


16 
12 
12 


28 

' 30 
31 


5 
6 
5 


1.78 
0^ts88 . 
1.85/ 


9. 


Enrichment with visit- 
ing lecturers, col- 
ioqula, etc. 


Researchers 
Teachers ■ 
Practitioners 


13 
8 
10 


. 32 
34 
30 


5 . 
9 


r, 

1.84 
1.98 
1.98 




*Total N*50. Rows may 


not add to 50 because respondents sometimes omitted an 


Item. 








■53- 

61 









CHARACTERISTICS OF PR0GRAK OPERATION 



• i 



Number of graduate deans who rated 
information about "the characteristic:* 



Characteristics 



In programs 
to train r 



Eaaen trial 



Important 

(2) 



Perhaps useful; 
' not important 
(3 and 4) 



Mean 
rating 



IQ, Support sjtaff for aca- 
demic program (secre- . < 
tarie8» research 
assistants, teachljAg « 

assistants, etc.) 

' > ^ ■• 

11. Provision for meeting 
individual needs of 
students 

12*. Personal ianS 

academic freedom 



13. Relationships with 
cognate programs 



14. Efficiency of degree 
production 



15- Siz^^f the ^program 



16. Plans for the future 
of the program 



17. Internships or other 

* opportunities- for rele- 
vant student experiences 

18. Postdoctoral study 
opportunities 



19. Recent innovations 



.... _ ^ . 

Researchie^i^s 


14 


Teachers ^ 
Practltldnefs 


12 


8 


Researchers 


14 


Teachers 


10 


Practitioners 


14 


Researchers 


18. 


Teachers 


19 


Practi^oners 


18 


Researchers 


9 


Teachers 


10 


Practitioners 


8 


Researchers 


7 


Teachers 


5 


Practitioners 


6 


Researchers 


8 


Teachers 


8 


Practitioners 


8 


Researchers 


11 


Teachers 


10 


Practitioners 


10 


Researchers 


1 A 

10 


Teachers 


21 


Practitioners 


23 


Researchers 


10 


Teachers 


1 


Practitioners 


1 


Researchers 


8 


Teachers 


7 


Practitioners 


7 













29 


7 


1.86 V 


27 


10 


1.96 


31 


10 


2.04 


25 


10 


1.92 


29 


• 9 


1.98 


24* 




1.92 


15 


15 


, .1-98 . . 


15 


13 


, 'J. ^ . , 


11 


18 




29 


12 


2 .06 ^ 


29 


10 


' 2.00 W 


27 


1<» 


2.12 


29 


9 


2.07 


30 


9 


2.11 


31 


7 


2.05 


30 


11 


2.12 


26 


15 


2.16 ; 


23 


17 


2.23 


23 


16 " 


2.14 ' 


24 > 


15 


2.14 


24 


wl5 


2.14 


21 


18. 


2.20 


23 




1.65 


21 




1.65 


19 


19 


2.21 


10 


36 


2.87 


13 


33 


2.83 


21 


19 


2.27 


27 ' 


t 13 


2.17 


24 ) 


16 


2.21 
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CHARACTERISTICS OP PROGRAM OPERATION 

9 



Nuaber of gradtiate deanB who rated 
infoymtion about ^he characteristic:^ 

Perhaps useful; 



Characteristics 



tn ^ro^rama;^, IBsiiehtlal Important: ^ hot important 
to train: (2) (3 and 4) 



20. Program dectilpn-making 
structpure - 



21 • Competitiveness (among 
:%tudents 



Researchers 6 

Teachers 4 

Practitioners 4 

Reseatchers 5 

Teachers 1 

Practitioners 2 




22 « 

24 ' 
22 

17 
19 



21 
27 
22 

27 
28 

31 



Mean 

rating- 



2.^1 
2.77 
2/38 

A 

2.79 
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Appendix C 



The Adequacy \^nd Availability of Measures that niight be Used 



as Indicators of a Program Chatracteristic *^ 



Notes 



Raters were asked to answer, the following questions^bout each program 
characteristic: ^ 



^. Q.l a) In your opinion,- how adequate or appropriate is the measure 
■ as ^n indicator of the quality of the listed program 

characteristic? Circle one letter for each measure, using . 
the following key: 

■^^ VG = very good indicator 

i! . G * good indicator 

P f poor indicator!, but useable 

,1 = inadequate indHpator ; not useable 

' b) From all of the possible measures for each characteristic, 

including any you may have added, double circle your rating 
of^ the one measure for, each characteristic that you would prefer 
to see used as an indi^c^tor .of program quality. In making th*ls 
^ judgment you may want to' coi^sider practical problems of gathering 
the information as well as it's more abstract jdesirability, 

Q.2 Is this information currently available through the 'program or 
dean's office at your university? Circle one lette?:. using th'e 
following key: y ' . ^ 

Y = yes, information could' be supplied from records without 

' new data collection 
N = no, information is not available at present 
* V = varies from program to program, or not* sure whether 
> available or not < 

Since jtidgments about the relative importance of the various characteristics 
in different kinds of programs were available from the earlier survey, this 
questionnaire did not ask respondents to make distinctions among programs 
emphasiEing th^^ preparation of research scholars, college teachers, or pro- 
fessional practitionets. 

1. Toual number of respondents is 43. Frequency tabulatj.ons C^pr ef erence" 
column; "adequacy*' and "availability" rows) may not add to 43 because of 
omissions w The omission rate was slightly higher on the "availability** 
question than on the "adequacy" question, particularly in the last section, 

I. Freqtiency tabulations for "other" measures indicate the number of persons 

who wrote in -some additional way to measurie the characteristic. Mean ratings 
and "availability" are not reported for these write-in ineasu?es. 

Program characteristics and possible measures appear in these tables exactly 
'--'as th^y appeared in the' second questionnaire; the only change is that tabu- ^ 
, lated, response frequencies ^i^eplace the questionnaire's response options. 



I 
I 



•TikUC.l '• ^ 
FACULTY ch«ritt(ristli:s and pOislble netiures . 


Preferred 




0.1 ' 
Ade(|uicy 

Rated 11: 




Hcan 


9.2 
ATllllMUCI 


■ ■ 


by; 


Very Good Good Poor or Inid. i^etlng 
(4) (31 (2 ind \) - 


Yd No VirlM 


I. Acidnic crtlTiliig of faculty 
















1. Percent with Ph.D/s or equivalent degrees 


U 


20 


19, 


4 


■ 3.37 


42 


' ' ' ■ t 
D, Perctn(j)f Pn,D« s iron^ top progrioiB in eich lield, as deiined by 
a reWatlonil survey 


18 


20 


18 


5 


■3.33 


! ' t ■ 

28 6 B 


^c. Mierige gra^s in graduate school 


0' 


1 


13 


29 


2.09 


7 25 ' 11 


d. Percent who held national fellowships as graduate atudents or poaY-; 
i doctoral fellowships since graduation 




6 




12 


2.B6 


11 11 20 


e. Appropriateness of specialty. areas of training to purposes of the 
program, as evaluated by a visiting teai^ oOxperts ^ 


k 


11 


22 


10 


3.00 


5 16; 19 


f. Percent of enrolled students who rate the academic achievement of, 
faculty nembers "excellent" 


2 ' 




16 


'21 


2.49 


2 28 12 


g. Other (write In) 


1 


2 










» ' - - - t 
— - —•— ■ ^ - - - ■ - ■ ^ ^ . 














(Oouble*circle one ineasure you prefer) ^ 


■• % 


i 

; ■ vi» 








> . 



2. ■ Research "(or^other. scholai'ly or creative) Activity 



er rTs 



a. Average number research* projects in progress per FTt facul^ty 

b. Average fjumber resenrcH proposals funded in the last three years 
per PTE "faculty 

c. Average percent of (acuity time spent in basic research (or 
^equivalent) 

d. Average percent of faculty tine spent in appl lad' research 
(or equivalent) .1 ^ 

e. Average nunber research assistants per FTE faculty 

f. Percent of faculty actively Involved in the,'piiblicatlon of research 
results (journal editor, editorial board neiaber, referee for sub* • 
Bitted articles, etc.) ^ 

g. -, Average number Invited presentations of research results in the 

past year (visiting lectures, colloquia, workshops, professional 
meetings, etc) per BE faculty 



h. Other: 
. f ■ 



1 Mill Ilk j" ^ m ' 



vi''', 

ii: 



ERIC 



(Doubie'drcU one mmt you prefer) 



14 ;/:;i9-"',, 
■ ■ ' . I' 

19 22 



10 ■ 02 



2 3.40 



• 23 
33 



9 10 , , 

4 .V';- 



6 22 15^ 2.10 



2 



20 
11 



20 
30 



2.53 
2,14 



15 24 4 3.26 



3.33 




■14 14 13 



U 
25 



15 14 
5 8 



21 10 9 



16 , 9 14 
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TACUin chiuctirlitlcB and poolble wiiurii (cont. 

_ • I ■■ — - ^ — - — 



3. Hiiiirch (or other scholirly or creative) [irgjuctlvUy 



I. ^ Avfrige nuiber booki iuthoretl end published In the hit threi 
yeiti per TU faculty 

,b. Avir||i nuibar edited bookii book chaptira^ or •onogripl)i 
piiiliiibid Ih tha U^t^threa yeari per FT^ faculty 

c, Avtraie auriiet Journal arttclea published in the last three 
yaari par Ttl faculty 

d. I|il||)tad average' nuiber publlcitlona (giving progressively ^ 
lore tftifht to refareed jourtiili, single authorship of articles, 
Jenlor authorship of bpoks» etc*) In the past three years (sub, 
no. patents or artistic perfonsinces or products where appropriate) 

If. ' Percent of faculty who have publlatied at least one iten in tke* 
' ^ lait year , 



f . Average publications citation rite over the past three years 

g, Amage nuiber technical pipers or project reports prepared 
^1 In the last three years 

V' Other: (write in) . " 



0 



(Doiibleclrclt one nuure you prefer) 



Preffrred 



0,1 
A(le((ui cy 

Rated it; 



Htm 



by!' Very Good Good Poor or Inid. UM% 
W 0)-.(2i njl) 



1 ' 



" H 20 ' 

Si . n 



4 J'.JS 
3 3.14 
I 1.51 



' 0,2 . 
s AvitlibtUty 

Ye'i No Virlii 



32 6 3 
'19 6 ', 7 



33 



35 6 . , 2. ; 3.74 



12 18'. lb 



13 



23 
21 

27 
1 



14 2.81 



29 10 2 



9 3.07 ■ 12 18 . 10 



) 



15 Ui 



n 11 » 



1 , 



4. UoRori and avardt; national professloni^ recognition 



1. 'r. V-VtV''"' 



• 1 

. kniift reputational rating of the faculty uong colleagues (cf* 
ACE ratings) 


13 


12 


25 


i 3.14 


16 13 


11 


b. Avira|e nuibtr professioDal hooora and avarda;per FTE faculty , 
ov«r the put three years ' 


17 


19 


19 


5 3,3) 


15 9 


17 


c. Amage nual)er professional aain, offices, (ioiilttee chalnan- 
ihipa^ traveling lectureships, etc., per FTt' faculty over the 
put three years 


5 


9 


23 


U 2.9J 


14 9 


17 


d. Otbar: (wcita Id) \ 


2 


I 


1 






















r<:. r.h ..' . / . 

' ;H(jli4l^1«le' phe B^a^^re you pre/er) 








i 

* 
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FACULTY characteristics and possible measures (cont.) 






Q.l 
Adequacy 

Rated as: 




Mean 


Q.2 

AvauaDiiity 




by; 


Very Good 

w 




Yea NO' VarlM 


5. Concern for stud^t developmentjind welfare 

— — i<t " ' ' — ' ■ ' ^ ' — " — " ^ 


(3) 


(2 and 1) 




• 






5 






' ■' 

a. Average student (or forner' student) rating on a scale neasuring 
faculty concern for student development and velfare, accessibility, 
advising skill, etc. 


25 


15 


23 


3.23 


3 25 15 


b, Percent of faculty tine spent in conferences or Informal contacts 
with studeiils 


2 




n 


15 




3 29 10 


c. Reputational Judgnents by colleagues 


B 


8 


22 , 


13 


2,86 


2 32 7- 


d. Avg. no. of posted office hours per week per FTH faculty 


0 


0 


1 


42 


1.56 


14 18 10 


e. Tendency for faculty tp see students as colleagues, apprentices, 
eiiployees, or students as rated by: ♦ 














1) ^tu^nts 


4 


; 9 


13 


U 


2,83 


2 26 , 5 . 


2) faculty uenbers ^ 


0 


4 


16 


15 


2.69 


!■ 28 '4 ■ 


.3) program chainnan 


1 


3 


18 


14 


2,63 


2 26 ji-' 


f, Other: 


0 " 






1 






(Double-clrc le one measure you prefer) ^ 


,1. 











♦ \ 


■■^^ ^ = 








■-— f—^' — ' — 




1 



a. Avg. ratings by students on a course and teacher evaluation fonn 



b, Avg, ratings by colleagues on a course and teacher evaluation form, 
with or without cUs|- visits 



c. Number of awards for good teaching In the past three years 

d. Presence of innovaffvri^ aching proceiures^. and/or outstanding 
instructional n^terials, as^^ijdged'^y: 

\I) students J . / 
^^pr^an chairman and /or graduate^ 
3) visiting panel of experts 

e. Increase in knowledge of ^udents, as deiuonstrated on course final 
exams or standardized u 




16 



11 
5 



2^ '^y' \^jt 

20 16 

11 2? 



3.10 

2.63 
2.31 



B, 24 
5 22 
13 . 20 



7 



15 



I'd 
15 
9 

19 



2.93 
2.69 
3.10 

2,63 



5 20 13 

4 22 12 

3 24 12 

3 27 8 
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9 10 23 

1 30 10 
22 10 9 




FAClUV chara':teri9tlc3 and possible measures (cont.) 



/ 



h Teaching experience 



a. Avg. no. of years of teaching experience 

^ b. >.Avg, no. of years of teaching experience at A^^^^ or , 
university level in appropriate area of specialty 

c. Age 

d, ; Other: ' . 



(Double-circle one measure you prefer) 



Preferred 



0.1 
A de^uacj 

Rated as: 



Mean 



by: Very Good Good Poor or inad. rating 
(k) (D. (2 and 1)_ ' 



I (1.2 

Availability 



Yes No Varies 

_L 




8. Involvement in program affairs 



a. FacuUy'wt'l^iifactlon''ftith infjdfe^^^^^ important decisions 
cbncernlng^|j)<ei progran .v' , y. ^ 

b. hKMfm, (H0|lti,.|j(tvlng on one or more program/if p* 



tuent 



'i ir ■■' 

jlit'^;hc\ilt^ ^j^tt df agreement with and connitnent to the purposes 
; ^ , V ind gbjyirot the program , 

)V - 4»?*"'liW turnover rate 



i. 04*r 



ible-circle, one measure you prefer) 



13 



27 3 3.21 



5 21 17 2.65 



24 
11 



|30 



3.26 
2.02 



3 25 15 

20 15 8 

2 28 13 

32 7 1 



9. Group Dorale or esprit 



71 



a. Avg, [j^(;ulty-reported satisfaction vlth program leaderships 
enthusiasm for the program, loyalty, Involvement, etc 

b. Lov turnover rate 

c. Relative vitality vs. malaise among the faculty, as rated by a 
visiting panel of experts ' 

/ >' 

d. Faulty sense of comity, feeling of shared Interests and 
Invdlve^nl Jn worthwhile activities^ as rated by: 

1) faculty menbers 



2)'prograni chairman ^ind/or graduate dean 



3) vl^tlng panel oi experts 



(continued) 



22 
0 



22 17 3 3.45 

1 ^ 15 27 2.16 



15 23 



17 20 



5 25 



11 23 



5 3.23 



4 

U 



3.32 



3.07 



4 22 13 

30 6 5 

4 24 12 

■ • 5 

2 25 11 

1 20 11 

3 23 12 



ERLC 



fALULii cniricicrisiics ano possiDic n^dsurcs ^coni>/ 


Preferred 




0.1 
Adequacy 

Ra^ed as: 




nean 


9.2 ' 
AviUablllty 




by: 


Very Good 
(4) 


Good Poor or Inad. f«in« 


Yes No Varies 






(3) 


[2 and 1) 






e. Avg. self-reported informal social interchange and camaraderie 
a&ong the faculty 


, 0 




18 


25 


2,26 




f, Other: 




I 










iDoublerclrcle one measure you prefer) 














10. Concern for undergraduate learning 


■ "a, Faculty-reported sensitivity to the interests, needs, and . 
aspirations of undergraduates 


■ '15 




20 , ' 


13 ' 


2.86 


1 29 10 


b, , Avg. percent of workload over past three years devoted to 
teaching undergraduates 


2 


k 


17 


22 


2.53 


26 8 6 


c. Percent of graduate faculty who teach at Uas't one' undergraduate 
course per year - ■ ' ■ 


6 


■8 


23 


12 


2,86 


29 6 ■ '5 


d. Avg. years of undergraduate teaching experience per FTE faculty 


0 


, .2 ■ 


iV^' 


27 


2.21 


■ 18 16 6 


e. Quality of a pFograni to train and super^Vi'se teaching fellovs,/ 
as rated by: , •■i '^* 




'„' ; ''u 




0 

, . ■ ■ i 

- ', ■ * 






I) students 


6 ' 






'v /' 




3 25 11 


2) faculty Benbers ' 


1 




.•31 


6 


■'2.95 '„:•■-:, 


1 26 12 


3) program chairman and/or grad dean 


0 




'.',28,' ■ 


12 \ 


2.68\:';;:3 


j 5 26 8 


4 . 

k) visiting panel of experts 




10 






2.93, ' 


1 2 29 B 


f. Other; 


1 


1 




If''' ■ ■ 






(Double-circle one measure you prefer) 














/■ 






.1 







I 




I 



Tii|tC.2 


K. ■ 

. Preferred, 




0.1 
Adejuac^ 

Rated as: 




Mean 


Q.2- 
Availabllicy 


STUDENT characteristics and possible measures . . 




Very Good 


Good Poor or load, ratlpu 
(3) (2 and 1) 


■ Us 


No Varies 


1. General academic ability of students entering the program 
















a. Avg. undergraduate GPA 


15 


18 


2^ 


) 


J. J/ 


'J 

34 


4 2 


b. Av|5, score on graduate aptitude teat (ORE, MAT, etc.) 


12 


21 




3 


3.42 ; ■. 


29 


3 9 


■ c, Avg. score on appropriate CRE Advanced Test (or other ach. test) 


3 


11 




i 

6 


3,09 


26 


4 11 


d..-Perrent of enroled students with national fellowships (NSF, 
' ' Woodrov Wilson, et^'H 


4 


12 


23 


B 


3.07 


28 


8 5 


e. Percent of entering ftudencs from highly selective undergraduate 
institutions 


5 


^ 9 


26 


8 


2.98 


23 


10 8 


f. Percent of enrolled students who rate the academic ability of 
their fellow students "excellent" 


2 


3 


19 


21 


2.51 


1 


36 3 


g. Other: , . 


1 


3 


: '1 










(Double-circle one measure you prefer) 


r 















2. Achleveioents, knowledge, and/or skills^of students at^tlme of" 
completion of degree 



a. Percent of graduates in the last three years who. published ^ 
somettlng prior to the degree ' ^ 

b. Percent of grad^jates iii the last three years who obtalfl^MSploy- 
ment directly related to their field of specialization 

c. Excellence or uni|ient?5S of dissertations and "theses in the 
last three years, as evaluated by external Judges 

> 

d. Avg, student score on a standardized test developed by experts 
in the field 

e. Avg. supervisor evaluation of internship/assistantehip performance 

f. Avg, student perfonnance on an oral exam at the end of the program 

g. Avg. perforiMnce on standardized licensing or other professional- 
level entrance exaias 



h. Percent of graduates In the last three years offered postdoctoral 
fcUowahipa 



1, Other: 



(DoubU-iijrcle one measure you prefer) 



10 16 21 5 3.24 
U 19 19 5 3.33 

11 23 17 3 3,47 

2 2 22 19 2,53 

1 ' 1 24 18 2,56 

1 5 20 18 2,60 

1 2 25 ■ 16 2.58 

1 8 22 13 2,77 



12 18 11 

20 10 . 11 

2 32 7 

1 34 6 

7 20 13 

15 16 9 

5 24 11 

15 13 U 
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1 

STUDENT char^cieristlcs and possible measures (cont.) 


Preferred 


■f 


Adejuacj 
Rated as; 




Hean 


Q.2 'A 
Avilhbillty > 

' ■ , . . !> J • . • 


ji''' .: . ■ , 


by: 


Very Good Good Poor or Inad. "tln|( 


Ye5. 


No " VMS 






W 


(3) 


(2 and 1) 








3. Calns^ln student knowledge and/or skills during their degree program 

, . 
















a: AVg, evaluation by faculty advisers and by asslstantahlp/lnternshlp 

supervisors of the "growth" observed In students over a tvo year 
' period 


n 


11 




6 


3.12 


B 


2? ] 


b. , Avg. student-reported score on a scale ieasurlng degree of satis- 
faction with gains In knowledge and/or skills during enrollment 


3 


5 


21 


1/ 


2.70 




34 S 


c. Avg. gains scores (pre-post) on a standardized test developed 
by experts in ttie field 


6 


3 


25 


15 


• 

2,6? 


^ 2 


34 5 


d. Avg, evaluation of galy In knowledge and skills by recent alumni 


11 


10 


14 


18 


2.6? 


1 


32 8 
t 


e* Other; 


1 


1 










In (Double-circle one aeasure you prefer) 














!'." 


■ ■ ' — — ^^^^ 

V Professional accoiupliihiints'of graduates 


, 1. Avg. no. pirt^licatlons by those who graduated in the last five years 


9 


18 


21 

■■'■V 


It 


3.30 


'4 


26 U 


b. Percent of graduates who have received one or more profes.siofial 
awards or honors In the past three years ^ 

> ^_ ^, 


2 


1'',' 


9 


3.02 ' 


5 


24 . U 


c* PerceiJt of graduates currently employed by doctorate-avardlng 
universities 


5 


13 


Ik . 


^ 6 


3.09 


14 


■ ■' 

■ i * 

15 U •■ 


d. Percent of graduates occupying positions of leadership and 
Influence lii the field, as judged by: 
















1) program chilruin 
Z) oulslde experts 


] 

10 


8 

21 


25 
1? 


10 
5 


2.91 
3.3? 


3 
1 


31 8 


e. Avg, income of graduates at specified intervals (perhaps 1, 5, 
10, «nd 20 years) 


0 


1 


U 


2B 


2.26 


0 


39 /'2 


f. Aruinl\ rating of satijhction with professional career perfonnance 
and developoent 


0 


6 


23 


13 


2.81 


0 


31 10 


g. Percent of graduates listed In "Who's Who" 


1 


5 


IS 


22 


2,49 


10 


28 3 


h. Avg, rating by employers on knowledge, skill, and performance 
of recent graduates 




9 


31 




3.14 


, 0 


' 33 8 


1. Avg. reputation among faculty at other universities 




11 


24 




3.03 


: 1 




], Other: 

— ■ .y — — — — , .—^-j, „ 














3^. 8 


(Double-clrcle one measure you prefer) 








* 









4 






0.1 
Ade(|uacy 




■Q.2 

• Ayiiliblllty 


Preferred 




Hated as; 


Mean 








STUDENT characteristics and possible measures (cont.) 


• by: 


Very Good 


Good Poor or Inad. "t^ng 
(J) (2andl) ? . 


Yes 


Ho 


Virlei , 


1 — ^ 

3. Career interests of students 










1 






a. Congruence of student career Interests with prograni purposes 
and emphases/ as judged by enrolled students 


19 


12 


23 




3 


30 


1 


b. Percent of students aiming for specified kinds and levels of 
employment, as estiniated by prograo chairman 


^ 2 




22 , 


16 2.56 




28 


9 


c. Self-reported career .plans of entering students 


0 




n 


21 .2.56 




30' 


4 


d. "Self-reported career plans of degree recipients 


* b 




25 


9 2.93 




23 


10 


e. Avg. student* score on a scale of professional involvement or 
comnltment 

' ] 


2 




16 


23 2.* 




37 


2 


f. Low student attrition rate 






12 , 


24 ^.33 


27 


10 


,3 


V 

e. Other: ^ 

• , 1 . ■ 






1 










(Double-circle one measure you prefer) 






•s 










6. Student perceptions of program q^ualiry ^ 

- _ . ■ ^ Lt- ■ ■ ■■ -M ■ 




a. Avg. student rating of the value of specified academic experiences 
in . the prSgram ^ ;i 




6 


31 


6 2.98/ 


3 

V 


3(1 


7 


b, Avg, alumni (1-5 years) rating of experiences In the progtam 




2] 


U 


2 . 3.58 


, 1 




6 


c. Ratio of inquiriea-^and applications for admission to number of 
students admitted etch year ' « 


5 


6 


19' ■ 


18 ; 2.65 ' 


28 


8 


4 


d, Other: 

(Double-circle one measure you prefer) ; 








1 » 








^- — — — - — • — : — 1 '-^ — - — ■ — — ■ 

7. Satisfaction wltji various aspects of t|[e degree program 








-...i: 








. r ^ 

a. Avg. no. complaints made to student complaint coDinittee 


0 


0 


1? 


31.,' ,2,105 


1. 


30 


4 


■ b. Percent of students who would 'reconsnend the program to others with 
similar Interests and abilities 


22, 


18 




1 3,40 1 

- ■ / 


2 


29 




■< 
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STUDENT characteristics and possible measures (cont/f^r 

-IT— — — -^i- 



r9 



Avg, student-rated satisfaction with'speclfled academic and . ,;;-v 
* non-acadeinlc Aspects of {he prograifl ^ *' ' 

Low student attrition rate 

t. Other: f 



(Double^cirde one measure you prefer) 



Croup wrale or ei 



iprlt 



a. Avg. student -reported score on a scale measuring satisfaction 
with program 

b. Avg. student satisfaction with fate of academic progress 

c. Avg, student perception of degree of support from vs. 
coBpiitltlon with other students In the program 

1 Student sense of comnunlty, , feeling of shared Interests and 
involveniert In worthwhile activities, as rated by: 

1) student; ^ 

2) faculty members 

3) prograa chairinan 

4) visiting panel of experts > 

e. Low student attrition rate 

f, Other: ■ • v 



(Double-circle one measure you prefer) 



Preferred toted as^; ' I ' 

by;' ' \firy Good Good Poor oT tnad'.' fstir? 
W ' (3) (2«nd( 



11 
3 
1 



,8 29 
2 -15 
1 



10 
■0 

0 
5, 



25 

21 



0 23 



6 

26 



3.02 
2.28 



[I- 



10 



20 



2; 86 

2,88' 

2.42 



I • Q.2 
Availability 

fn No Varies 



« 28 '8 
24 11 6 



3 ,31 « 
« 3 ' 27 ^ ' 6, 

1 32 \ 



23 


3 


3,30. ■' 


'4 


28 


7 






2,72 


4 


27 


8 


23 


17 


2,60 


5 


26 


. 8 


ID 


18 


2,50 




30 


5 


13 


Ik 


2,18 


18 


10 


■7 



Si 



f Tiblt C.} 

;. RESOURCES ind possible leasures 

' ,. J. , .. - 



1* UnlviriU«;iihanclal support for program 




I 



*' '^J^'M^P^^K^" ^"^8^^ allocation to total university allocation 

■ x'- ' 

b, Median faculty salary by rank 

c, '. ^ flUcatlon and general expense budget per FTE student ■ 

ij^ University support as a peccent of total program Jbudget 

e. JudgnenM'UbOut tile adequacy of university support rejatlvc'to 
program ])urposei by a visiting teap of experts 

4' Other: 



Preferred 



Rated as: 



Mean 



by: ' Very Cood-t Good Plor oV'Inad. rating 
' , (0 ' (3) ^ (2 and 1) 



(Double-clrcU one measure you prefer) 
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2.66 



'3,00 
2.90 



0,2 

AvatUblUty 
■Yes No * V«ti«< ;i, 



n u 5 

35 2 1 

30 "6 i 

32 3 i 

■'"■i ' : * 

,v5 ;20 11 




a.- Ade(iua|cy of relevant holdings as Judged, by: ^ 

1) visiting team of experts 

'J 

2) faculty members 

3) students ' ^ 
* 4) librarians 

f ^ i) program chalnwa . > 
■ b. Avg. no. books In the field per FTE student 
c: • Program library budget per FTE student 

d. Avg. ^elf-reported ,USB of the library by students and faculty 
members i. - * 

e. Avg. no. iooks and other laaterials checked out per FTE student 
Ifflthe past sitwnths^ 

f . >Exl|tence of other accessible libraries ift^fh^ vicinity, as - 
reported by prograi^^alnnan ' 

/ g. Other: 



0-83 



(Double-Hrcle one measute you prefer) 



19 
11 



22 17 
20 20 



30 
30. 



10 26 
2 19 
5 27 



3 
2 
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7 
6 
21 
10 



29 



\ 



22 



V 



18 



3.45 

3.« 

2^93 

2.95' 
'3!07 , 

2,«5 
^.81 

2.36 ' 
2,57 



9 19 

12 16 12, 

- 6 25 9 ■ 

10 21 J 
IB 17- 5 
23 13 k 



5 28 5 
11 20, 7 
^16 U 8 



RESdDRCES and ^-ossibU msum {roni,) 



3. UboMtoiy cqulpnent nd fi^llUiea (including facllltlel 'for thc'i 
creative arts) ' * - / ^ 



a. Square feet of Ubdtalfory and/or studla space pe^ PTE studmc 

b. Preience of laboi^ory/creatlve equlpnent considered essential 
or.laportaijt. by experts in thi field v ' ; | / 



c. Ade<iKjcy'drvljaboratory equlpnent and faclfltles as rated by: 



" " 3) vlaltljB teao of expert's 

,'■ d. 'Existence o/^ther accessible facilities in the vicinity, ny' 
'■: .^itported by program chalnun 





Ir: 



(Double-circle one measure you prefer) 



^ 4, Conputif ficWM^s ' 

1 \ ^ - ■ 



t ' ' Adequa cy 

Preferred Hited as; ^ Mean 

by: ' jVery Good .(Hood Poor or Inad. 



-a.' ^Adlfiacy 'of coap,uter ■facilities for needs of the prograti as 

rated by: . . ^ , ' ^ . ; 



1) program chalrnan^ 

2) facultyAnberi 

3) atudtn^ | 



4) vlaltlAg tein of experts 




b. Adequacy of software and con'puter. 5ipporf\irvfces to ncet needs 
of the program as rated by: 

1) prograv (Rinnan ' ^ 

2) faculty neibers 



c. Prograi budget (iwney afi(f/or tine) i 
per PTE student 




iputer 



faciflti 



es 



(continued)? 
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3.40 



2.9 

( 2-51' 
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"27 6: •■^1 
11- 13'. \1 



10 19 il 

^ 5 25 • llf 

7 22j =U, 

. Li 'l6 9 
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3,37^,' 
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18 
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3,12 
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10 
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3,44. ■ 
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10 * 2» 



*20 21 



12 20 



4 

2 

u' 



'3,12 
3,42 

2.95 



12 17 10 • 
9 19 11 



5 



8 



4 

■ ■ i 

AMiA/Rbu ina puiiAvic Hiiuici ^cunii/ t . 


'ittktrti H*""! M- , Hein 

IflAt Varu' f!AA^ /!AAil PAAr fit lilt 

RMf Tfry uow WOO roof or uniui ^^^-a'^s . 

* £ ,(4) t 0) (Midi) 


. 

AvilliUUty , 

lei NO viriii 


• •'• . ^ • — ~ 

d. blitence of other icceulMe coij^uterr In the vlcj^nlcy, u 
reporCid by pro|r«i cmmn ' ' 

y * ■ ■ ' , 

e. Othtr: " i , . 


'.1 25 17 v2.56 

' 


17' 14 9 


(Doubli-cltcle one ieasure you prefer) ' ; 

^ , , 



a. ^vi.' dolhr incoae per PTE faculty frov ot)tJ^t(leT^^ ^ver the- 
piit three years . ' ■ , 



b. .Ouli^iie funis as a percent of totll'^progrii^badjief 



tll'» prosrii hadaet 1^ 



c. Dollar »unt of federal research praject,^rant^ and contracts 
over the past three- years ^ ' 



d. Othe^:' 



(Double-circle one measure yoi) prefer) 



' 6. Financial support lor student^ , - ^ . ' 

a. Ayg. dollars for Sll fonns of financial ^8sistance per FTE stiidenj^ 

. , b. ^rcij|t^^f students reJeivlng any form of ^tjiang^l, ass^sUnce 

\ c. lUtlo.of students on fellowships or jrants to t'hj^^^ 

assl^antshlps • . , 4" :f. /^ ;f \. 

..d. Iirceot of students who hold fel^owslii^s orira^ft/from external 
•Ottrces i' / 




e. lucent of students or spouses assist^i|h obtaining ippjoprlate 
I .rf|)loyiBWt to support graduate study fc'V' " 



f. Pftceniof 



If 



stulf^ts wortlnl^ dissertaLor 



(Ons full-time 



^(ible-cirde one measure you prefer) 



12 26 



5 



IB. K 

► ■ 



5 ,11 23 



r 11 

5 17 
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9 

31 
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2, '98 

2.1i 
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5 . 22- . 19 
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RESOURCES and posBlbh oeasures (cont. 



■ 7, Furpoies and atrcngtha of parent iwtlc\jtl6n' 



Riputation of university as rated by knovledgeahle educators 
. ;b; Huber of doctoral prograia In related fields vlthin the'unlversity 
f^: Cohiruence of progru and m^viralty purposes as Judged by:. 
/ — 1) vlalting {janal of experts * ' 

2) prograi chalraan^ndM graduate dean 



J. 1 



M 
I 



faculty leiber^ 

4)v coll/aguia ll| slBilar.prograss at other universities 

d. level of support for the prograa from central* university 
^adilalstrators, as Judged by: 

1) viiitlng panel/of exptt^ 
\ I) prograi chalrUQ and/or gfaduate dean • 
, 3) fscult|.iieBbe)rj 



Preferred 



Rated as: 



Hean 



by: Very Good Cooi Poor or Inad, i^^H 
(4) (3) (2 and I) 



Level of ^dBtssions selectivity in the Jbge; university 
nlative tft the progran under study W 

f. Other: ' 



{Double-circle one aeasure you Ipref er) 



u 

3 

6 

3 
3 
0 



22 20 1 

8 24 11 

IJ 22 , 5 

6 2ft >'7''^ 



4 



5 ,25 12 



3.49 
2.81 

'3.19 
2.88 
2.86 
2,79 



15 , 19 



q.2 

AVilliblUty 
Ym No ' Virlci 



9 ' ' 3,07 



9 9 



1 % 4 ''m*^ 



11 



23 U 



"'2,93 
2,79 

2,67 



9 18 13 

34 3 3 

7 17 ,13 

4 16 ' , } 

8, 19 10 

/l''.25 11 



5 20 : (3 
15 15 I 

■*8. 20 ' 40 



y ' 9, Cluirofli ind, office, jpjce_fot the progran 



' '.0' "'15^' 



instructional spsct pet nifUjjdent 



b, » Avg; square feet of office and resesrch s 

c, Pircwt of faculty itobers In single-person rffkes 



i. ^ Adequacy of classrooe anS office space as rated by:^ 



, 1) .students 

2) faculty neabert . ' 

3) progran chaltAnd/or graduate dean] 

4) vliftlng ten of'exp^rt* " _J 
e. Other: 



^ ' (Double-circle one Beasure you prefer) , * 




' ^ 5' 
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24 
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27 




2J1 


26 
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X 


9 


1 
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5 


26 


8 




' 30 
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19 


*14 


12 


'.22 




3,05- 


.A 


29r, 





_ 1 



■v-v 



» ""OfEllATrOllMrfliaricterUtics and posslblt measures , 



o^e program 



■4^ 



4 S 

I 



a, CliHly of program i^^ws and plans, as judged by: 
I) 'faculty i)ietiib<?rs ^ ^ ' 

. 2) program chairman and/or graduate dean ' 
3) visiting pinel of experts ^ 
M students , 
5) recent graduates 

b. Ratio of graduate to undergraduafe students In the program 
c\^atlo of graduate to undergraduate degrees awardfd 

d, Relative emphasis on research^ 4,.eachinj,,and^iervlce, as judged by: 
1) faculty members : . 
I) program chainnan and/or 
3) vlsltlng^eam of experts 

e, Number of areas of specialization vitbln the 

f. Ratio of part-time to fall-time students enr/ued 

g. Other: 



Preferred' »^:\yted as: Mean, 
byi VetiiGQpd' ';Good Poor or' Inad". ratlnf 



T 



2. Adjlsslons policies ^ 




gM3) (2 and 1) 



AvtlUblUty 
Vei N9 Varies 
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3.26 


1 
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9 


31 


1.95 ' . 
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2.95 


12 


20 
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't . 
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.2,98 


. 12 
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19 
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1 


12 


22 ■ 
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3.05 ■ 
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13 
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2 


11^ 


29 


. 2.21 


29 


5 


5 


0 




13 


'26 


2.28 


27' 


5 


6 



a;Arcent ofj qiialifiid,applkants'ifi'o;ar> adinitted^^ reported 
^w^y the ad^ssions, iomiittee 

' b. Rijiatlve se|^tivity of graduate and tndej 
■\ as reported by the admissions cotnniittee 

c, Annu*! recruiting and ^dijissjons processing cost per fjl student 




,,1.; 



■3 



I 

I i 



4 . 



w 



lontinued) 





'11; ^l'' 



U%2.3|./.4' 



00 



OPEIuflOMAL chifKUrlaClcs and possible neaiores (cont.) 



d. Judgnenc of whethef adQls'alons standards should be higher or 
lover, as rated 'by; 



.... . i 




1) students i^'^ftf 'v' 



3) progras chalrun 

4) .vliltlng panel of/^^^^te 

.i'^*^ e. Clarity of adilaalons procedures and. standards, as judged by; 
1 




1) visiting panel of experts 

2) itudtnts 

3) prospective students 
f, Other: 



(Double-circle one leasure you prefer) 



3. Provision for the evaluation of student progress 



I 
I 



a. Clarity of requirements and standards for progression frosi 
entrance' to candidacy vlth stated tines for review and 
evaluation, as judged by: 

i. 

1) prograi chairnan 

2) students ' ' 

3) visiting panel of experts 

b. Student^reported^ snisfaction ^jJBLheir prnressJIf^ th^ ileg^e 

MM ^ . .^Hb. 'I . i: 



Preferred htti u:' Ncm 

by: Very Cood Good Poor or Inid. "'ing 
U) (}) (2indl)' 



1 

4 
2 

12' 




I- 



c. Quillty ind regularity of r«poil 
' lupenliort, u Judged byt 

1) tflil,clag pinel of expertt 
vZJ fiiuUy advliers 



m 4 

(||tetnship/a 



Intship 



^ , L Percent of students I'ho spend 18 Aonths or more In fulhtlse 
enrollient (Including assistantshlps/internshlps) 

^. Distribution of grades assigned to doctoral student's ' 



ERIC 



f . • Other; _ , , . 

(Double-circle one measure, you prefer) 



f 
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11 '24 
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Yei 'Mo V^rlM 
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OPEMTIONAL chirirterliclc^ ind p(|»Blble mMsures (cone. 



i Couric iitd progruofferlnKs 



a. Courses ^pd olhijrivUlable experiences appropriate to purposes 
of the prograiand specialty training of the faculty, w Judged byi 

, 1) faculty Mbera 
' . 2) 'itudeDti . 

3) vlaicing panel of eiiperts 

b. Percent of totil student credit Mrs produced by s^lnirs and 
tutorials ^ 

^ c. Avg. student ratings on tha virlet^ and relevance of cou'rae ■ 
offerings ^ 

d. No. of new or cowplelely revised coiiraes Ip the Iiit tvo years 

* 

r Ease with whlck new tourses can be Introduced » as judged by 
faculty «e»kera •* , 



f. Relative need for curriculuw review and updating as judged by a 
' vliltlng panel of experts 

g. Other: 



(Double-clfcle one •easure you prefer) 



J. Progrtt leadership and dedslon-niaklng 



a. Quality, of leaderahlp provided by the progran chalnwn as 
judged by: ^ , 

^\ 1) faculty oeiberi 

2) gradi^te dean ^ 
^ 3) visiting panel of Aperts 

b. Faculty seU-reported activity and Influence' In the larger 
unlye^raity cowjunity 

c. totentlof deWratic or decentralixcd declslon-naklng in the 
pfogr*«Las n^ed by: ^ ' 

1) facuUty mm} 
I) stud/nts • 

f ^ \ 

3) pp>^raiii chalrnan* 
ji) visiting panerfif experts 

(continual 




T V ,■ . 2.63. 
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OPERAIIONAL characterlflt [C8 and posslbU measures (oont.) 
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■ .4 

Adequacy 






AvilUblllCy 


Preferred 


Rated as: 


f 


HMD , 








d. Satisfaction with opportiinUy to Influence decisions us rated by: 
■ 1) faculty irieinberB 


'by: Very Good Good Poor or Inad. f*tlni! 
W (3) (2 and 1) 
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>o 
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2 

• f ^ 
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9 ' 


i t 7U 
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e. Other: 
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(Double-circle one measure you prefer) 
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6. Provision for the weffare of faculty members 
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t 

a. Percent of faculty with the rank of professor and assbclate 
professor ' s 
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b. Percent of faculty with tenure 
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c. Median salary by rank 
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3.26 
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d. Excellence of fringe benefits package, as judged by; 
















1) faculty meabers , 


* 

n 


Q 71 

7 iJ 




0 01 
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19 
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2) outside experts 


l 


10 27 




3.10. ' 


7 


26 
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e. Number of faculty meabers who left and number who have. been 
replaced in the last three years js a percent of total number 
of graduate faculty 
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5 15 


22 ' 


2,45 ' 


70 
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f. Provision for assistance to new and young faculty, as Judged by: 


■ 








^ 






^ 1) faculty nemberg 
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9 26 




3,00 
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2) visiting panel of experts 
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9 27, 




3.07 
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g. Faculty satisfaction with the quantity and quality of support staff 
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2.91 
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22 


13 


h. Faculty aatirSfaction with' freedom to plan courses and conduct 
research without Internal or external interference 
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12 24 . 




3.07 


4 


24 


11 


1. Fdculty satisfaction with freedom to pursue personal life without 
internal or extfernal Interference 


~ 0 
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U 


2.91 
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25 


• 9 


J. Extent of department and university support for the principle of 
acadetBlc freedom, as Judged by outside experts 
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L 
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k. Average teaching load I 
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(Doublt-drch one neasure you prefer) 
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OPERATIOHAI. fhificterlitlcfl and possible measures (cont, 



' /. Job placement of graduates 



a. Satisfaction vttli program efforts to place gradwteSp as Judged by 

1) students 
• 2V faculty nembers 

J) vlsltlt^i; panel of experts' 

'b. Percent of degree recipients in last three years who were placed 
In posltl(ws directly relevant to their graduatf education 

'f. Pe.rrpnt of recent graduates who Joined faculties at uajor 
doctorate-producing universities 

d. Profile of jobs taken by former students and graduates In the 
last three years 

e. -Other: 



(Double-circle m moasijre you prefer) 



fl. Provision for the adviseraeni of students 



a. Qiullty of th> advising system, as rated by: 

1) students 

2) faculty members 

3) visiting panel of experts 

4) recent graduates 

b. Avg. no. advlscpB per faculty adviser 

c. Avg. no. dissertations or theses directed by each faculty member/ 

d. Other: 



(Double-circle one measure you prefer) 



Preferred 



Adc^mcy 
Rated la: 



by: ' Very{k)od Good Poor or Inad» f«tlng 
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OPERATIWAL ch«racterlatlC8 and |)dl|ilble leaaurea (cont. 



Preferred 



Hiced m; 



Mein 



9. Student -faculty kteraction 
> ' " " " 



by: Very Good Good Poor or Ined. ritlng 



(4 



a, 


Avg. no, atudent-re|yorted Informal contacts with faculty members 
In 1 typical semester 
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1 

7 21 


15 


2'.70 


1 


32 
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Avg. no. faculty-reported individual conferences with students 
in a typical week ' ^ . 


1 ^ 


9 18 


li 
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30 
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c. 


i 

Avg. no. faculty-reported hours per week available to students 
for questions or consultation 


i 
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5 U 


24 


2.47 
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'26 
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d. 


No. students and faculty nesbcrs who serve oi) Joint student-faculty 
conlttees 

1 " * 
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13 


2.fO 
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13 


10 


e. 


Avg. faculty and student scores oiia scale Mlturing the;relative 
facdty perception of students w colleagues. ap;irentlc^s:,i 
eoployees or students 
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21 : ■ ■ 
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f. 


Student-reported satisfaction with opportunity to work closely 
with at least one nettber of the faculty ^ , " 
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3.42 " 
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28 
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Provision for informal lounge space used Jointly by students and 
faculty 
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Other: 

(Double-circle one teasure you prefer) ^ 
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(3) (2 and 1) 



'q.z 

AvaiUblllty 
Yes No Varlea 



^ 10. Degree requirements 



a. PlexlblUty of program requirements sufficient to meet Individual 
student needs, aa Judged by: 

1) studenti \ 

2) faculty meters 

3) vlaitiog panel of experts 

b. Perceflt of students who undertake a research project (other than 
dliiertatlon) 

c. Percent of students who serve an apprenticeship or asslstantship 
(appropriate area of professiond specthlty or teaching) 

d. Percent of th^ program required In full-time residence 

e. Percent of total requirements (credit hours or time) specified 
la a coaoA core of, courses and experiences 
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(continued) 
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OPERATIONAL chancttrlstlcs and possible measures (cont. 



f. Avg. percent of annuiil first-year students who are dropped fdr 
inadequate ^cholarshiip or refused advancement to candidacy within 
four years (or equivalent) 

Clarity of 'specified competencies and qualities e)cpected of 
graduates preparing to be researchers, teachers, or profeaslonal 
■ practitioners, as evaluated by: 

* 

1) faculty neflbers 

2) students 

3) visiting panel of experts 

h. Percent of studeijps on Individually planned programs each year 



I. Other: 



(pQuble-clrc'le^one measure yoij prefer) 



Q.l 
Adeq uacy 

Preferred ^^^^ 
by: , Ver) Good Good Poor or Irtid. ^*tlin 



11. EnrichnenC with visiting lecturers, colloqula, etc. 




Q.2 

AvilUbllltY 
Yii lit VarlM 



• • ■ r " ■ " •■ — — — ' 

a. No. visiting lecturers, colloqula^ etc., scheduled In the 
past six months v 




20 


23 • 


> 

15 


4 


.3.43 




7 




b. Avg. > aft enhance per scheduled lecture, colloquium, etc. 

c. Satisfaction wit|^ enrichment efforts of the'^program as rated by: 


\ 


2 


8 


21 


13 


2,81 


4 


25 


10 


1) students 




4 


12 


24 


6 


, 3.12 


3 


22 


12 


2) faculty itcmbers . 




2 ' 




26 


5 


3.12 


6 


21 


10 


3) visiting panel of experts 




8' 


13 


20 




3.13 


2 


23 


12 


d. Other: . 




I 


1 














(Double*clrcle one measure you prefer) ^ 




















12. Relationships with cognate prograna 










■ nil n 










^, ^ — ^ — ^ , — « — — .-^-^-^^^ — — - - 

a. Percent of program students enrolled for one of nore courses in 
another department 

b. Percent of students registered for an interdepartmental program 


\ 


U 

1 


'14 
6 


22 
22 


6 

13 


' 3,14 
2.78 


22 
20 


U 

11 . 


7 
9 


^ (cqntlnuedj 
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194 



OfEMTIONAL chime terlrt lea ^fi'd possible meafures (cone) 



t. Rdatlonahlps and interchang«a vlth cbgnate prograss aa nted by: 
I 1) faculty Mftbers , 
2)4'tudenta 

})* progra chilnun, and/or graduate d(an< 
, 4) chilrten of cognitc depahMnts ^ 
d. Pfrcint of coursci that ar« cro^a-listed with another depittncnt 



4., Avg: ftudent credit hours In prograa coufsea each seoieateT that 
ire produced by atudenta enrolled In other prograns 



f. Other: 



(Double-circlVolietBeaiure you prefer) 



13. Kft^iency of dtegreeVoductlon 



Preftrrid 



Adejuicjf 
Rated is: 



Hein 



by: Very Good Good Poor ot Inid. siting 
(M (3) (^fliidl) , 



31 
29 
30 
32 
U 



11 12 



2.82 



I 

00 

0 . 

I 



^Ire of the progrn 



Q.2 

NUihilU| 
Yei No Virl«( 





8 


22 


10 


3.05 


6 




10' 


3.10 


10 


20- 


10 










3.05 


9 


19 


10 


2.57 


26 


6 


7 



18 17 ! 





- - - - ^ .^^^ , 

Percent of thoae who enroll who «a^ the Ph.D. ' 


18 


. 21 20 




3,45 


31 


6 


3 


■ ,v 


Percent of thoae who enroll who achieve candldacV 


0 


6 24 


12 


2.76 


28 


8 


4 




HWlan nmber quarters or sewaters In attendance that are needed 
to cotplete the degree 


5 


9 23 ' 


10 


2.86 


24 


t 


8 




Eatiiuited total avg. tine required to achieve candidacy 


0 


5 28 


9 


2.83 • 


21 


10 


8 


1 e. 


-Estluted total avg. tlee required to coinplete the degree 




11 27 




3.U 


25 


8 


6 ;. 


.f. 


No. of degrees awarded In each of the past four years 


1 


4 ^19 




2.55 ' 


") 


■1 ^ 


. 2 


. (. 


lUtlo of doctorates awarded to no. of graduate faculty an4 to 
DO. of ear oiled students' ' \ 


10 


12 21 


8 


3.02 


■ 28 


6 


5 


h, 


Other: • ; ^ 


0 














.r 


'(Doublf-cirde one aeasure you prefer) 




/ 










1 



/ . ■ ' — — . — 1 

a. No. of enrolled students and first-year students In each of the 
past four years, by full-tine and part-time 


20 


19 . 19 4 


3.31 


36 2 • 2 


b. No, of postdoctoral students enrolled in each of the past four years 


1 


3 16 ' 23 ■ 


2.36 


29 '7 4 


(continued) 








r 



I 

I 



li niii , I 



, ^„ I'f Ht*»M«^ Ui^jjy 1, ,(^„r yMi'i. 

„„<lnii/lAnil'V t«ii„ 

II,, |,„„r,™ diKM-'"''"' 

/. ml'''' 



I 



ir'«"'"KP""i "I ,,s \ 

ti. ml"' 



0.1 



Prifirrf(j M(»d m: N««ri 



U 30 
? 1(] 



?9 i.n 



q,2 . , 

n 5 I 







* 




1 






9 




(>. ),02 


11 U 


10 








^ 1,01 


S H 


13 


20 


Ifl 


?() 


1 , 1.1J 


,^ ^0 ' 


\s 



f 



pre"*'"''"*"' '"'I" a„|^„„,n, In ih. rn>ir«' th-l, if 
tl»M,,^ ,„r,.ll«'"l I" IW„ 

p^f^-nl "I r'«»nd, ,,,,i,t.,„„ who Mr tn.llvl<l»*l 
„l,«rit(ifv. ot c«nt,r ' I 



o 1)7 



.JO 



II V i> ■ \M 
11 « • 1.U 



11 Ik 



2 It 24> 



U 2k 1 



I.Of 



1.11 



2.6(1 



10 •> « 

10 ^ 4 

20- U ■ «■ 

fi *'2 II 

10. 11 ^ 



Appendix D 

Ranked Importance of Major Tasks of Ph.D. Programs 
in Fifty-four Universities 

> 

Graduate deans in the sample were asked to rank the major tasks 
of their own institutions' Ph. D .- programs in the order of their 
importance in the physical sciences, biological sciences, social 
sciences, and humanities. In general, training research scholars 
ranked' first in the physical and biological sciences, training 
college teachers ranked first in the humanities, and training research 
scholars and college teachers received about equal weight in the 
social sciences. Training professional practitioners ranked third 
in all four curricular areas, highest in the social sciences and 
lowest in the humanities. Frequency tabulations and mean ranks are 
presented in Table D.l. 

RaaCiks assigned to each task in each curricular area were also 
examined for differences de;pending upon the respondent''^ field of 
specialization or the characteristics of the universities represented 
in the sample. Size of graduate enrollment was not a factor, but 
deans employed by private and high-prestige universities reported 
more emphasis on the training of research scholars in all four areas, 
while representatives qf public and non-prestige universities gave 
slightly higher ratings to the training of college teachers and pro- 
fessional practitioners in all areas. Deans trained in the social 
sciences and humanities reported that the physical and biological 
science departments in their universities strongly emphasized the 
training of researchers, while deans who were trained as scientists 
gave this goal somewhat lower ratings for science departments in 
their universities. Field of training was less related- to ratings 
assigned tasks in humanities or social science programs. In all 
cases, however, the training of researchers was perceived to dominate 
the physical and biological sciences while the preparation of 
teachers was most important in the humanities. Training profes- 
sional practitioners received lower ratings by all subgroups of 
respondents in each curricular area with most endorsement in the 
social sciences by deans in public institutions. 

pifferent tasks for different fields of study or disciplines, as 
well/ as among programs within disciplines, suggests an additional 
dimension for consideration in ^the development of systematic pro- 
cedures to assess quality in doctoral education. 
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Table D.l 

Ranked Importance of Major Tasks of Ph.D. Programs 
in Fifty-four Universities 



Number of deans who Mean 
ranked each task:* rank 

#1 n »3 

Physical sciences major task is' 



Biological 



to train research scholars • 


32 


11 


2 


1.33 


to train college teachers 


9 


21 


15 


2.13 


to train professional practitioners 


4 


13 


28 


2.53 


sciences major task is 










to train research scholars 


, 28 


14 


3 


1.44 


to train college teachers 


14 


24 


7 


1.84 


to train professional practitioners 


3 


■ 7 


35 


2.71 



Social sciences major task is 



to 


train 


research scholars 


18 


15 


12 


1.87 


to 


train 


college teachers 


18 


18 


9 


1.80 


to 


train 


professional practitioners 


9 


12 


24 


2.33 



Humanities major task is 



to 


train 


research scholars 


10 


25 


6 


1. 


90 


to 


train 


college teachers 


29 


12 


1 


1. 


33 


to 


train 


professional practitioners 


3 


4 


34 


2. 


76 



*Rows do not add to 54 because a few respondents did not complete this 
part of the questionnaire and some universities do n6t offer Ph.D.s in all 
curricular^ areas . 



Curricular Area 



Table 0.-2 

Major Tasks of Ph.D. Program^, as Ranked 
by Graduate Deans in 54 Programs 



Curricular 

Area 



Programs 
to train: 



Average rank order of importance 



Physical 
sciences 



Researchers 

Teachers 

Practitioners 



2.13 



2.53 



1.33 



Biological 
sciences 



Researchers 


1.44 


Teachers 


1.84 




Practitioners 









Social 
sciences 



Researchers 


1.87 




Teachers 


1.80 


Practitioners 


2.33 





iluoanlties 



Researchers 


1.90 




Teachers 


1.33 


Practitioners 


2.76 





Question: How wouia you rank the major tasks of your own institution's Ph.D. progrartis, 
in order of their importance? Tasics: training -research scholars, training 
college teachers, training professional practitioners. 



Northeast. . 

Boston University 
Brandeis University 
Brown University 
University of Maine 
Northeastern University 
Tufts University 
The University of Vermont 
Yale University 



East 



Appendix E 
Universities in the Safaple 
SoutI 



Carnegie-Mellon University 
Catholic University of America 
Cornell University 
University of Delaware 
Howard University 

Indiana University of Pennsylvania 

University of Maryland 

New York University 

Princeton University 

University of Rochester 

Rutgers University 

SUNY at Buffalo 

SUNY at Stony Brook 



Midwest 

Case Western Reserve University 
Indiana State University 
University of Iowa 
Loyola University 
University of Michigan 
University of Minnesota 
University of Missojiiri at Rolla 
University of North Dakota 
University of Notre Dame 
Ohio State University 
Southern Illinois University 
Wayne State University 
University of Wisconsin 



Ged 



Duki University 
Emory University 
Uni/verslty of Florida 

gia Institute <5f Technology 
versity. of Louisville 
phis State University 
iversity of North Carolina 
at Chapel Hill 
ahbma State University 
ice U'niversity 

niversity of Texas at Austin 
exas Woman' s University 
/Tulane- University 
Vandprbilt University 
University of Virginia 



West 



Arizona State University 
University of California — Berkeley 
Colorado State University 
University of Denver 
IJniversity of Hawaii at Manoa 
Idaho State University 
University . of New Mexico 
University of Southern California 
Stanford University 
University of Utah 
Oniversity of Washington 
University of Wyoming 
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